Proposed Improvements to Libraries and Overrides (based on "Links & Library Overrides are still glitchy after 4 years of development")
These are suggestions mostly from the thread "Links & Library Overrides are still glitchy after 4 years of development" started by @eobet: blenderartists.org/t/links-library-overrides-are-still-glitchy-after-4-years-of-development
This post (which isn't comprehensive - see OP) contains suggestions based on that, and other points raised in that thread:
Current Status of Overrides: Overrides work well for armatures, but if you go beyond that, you quickly run into limitations. The standard solution for these problems are "don't use overrides or libraries, just make it local!" But this really limits the usefulness of having libraries and overrides in the first place. Additionally there are glitches and bugs, and the UX is unclear.
Libraries and overrides are essential for larger animation projects.
I do larger animation projects on Blender, and for the most part, Blender copes very well. Blender certainly has shortcomings, some of which are annoying because they really slow you down (cloth, physics, LOD, bugs/crashes, viewport vs. render -> lost time = lost money) ... but they're not showstoppers: You can work around them. But libraries and overrides are not. They're essential to large projects, because you're using the same asset in many different scenes (hundreds), and you need to construct/navigate/maintain large scenes. I'm not talking about Disney-big scenes (like Sanfrantokyo in Big Hero 6), but still much larger than your see in your typical Blender short video.
For these larger animation projects, "make it local" just isn't an option. You need libraries and overrides. But the current implementation of these is causing a lot of pain. I was glad to see @eobet's thread, because until then I thought we were the only ones foolish enough to be trying this in Blender. But from that thread, it's apparent others using libraries and overrides are feeling the same pain.
This is a chicken and egg problem. From discussions on Blender Artists, it seems very few use Blender for larger animation projects, so change requests and bug reports have been very few in number, which might give BF the impression that everything is rosy. But it's not. If there's one thing that is keeping professional studios away from Blender, certainly the only thing that would ever drive me away from Blender, it's libraries and overrides. The rest I can live/work with/love.
Some posters on the thread have raised future functionality which might overcome some of these problems: Geometry nodes, USD. That's great, but it's pie-in-the-sky stuff in the distant future which might never happen. (Remember Animation 2020? Animation 2022? Now Animation 2025...)
But here's the thing: Libraries and Overrides might not win any beauty contests, but they're not far off either. I'd estimate they're 80% there. I think it'd be best to complete that, and if GN and USD ever deliver, great, but we need a solution here and now.
Why BF should do this: The more reliable funding sources are those who are financially tied to Blender's long-term success. But of BF's corporate funders, AFAICT Epic is the only one with a long-term interest in Blender (because as amazing as Unreal is, it can't do the type of DCC that Blender can). There's been much (very speculative) talk about why professional studios aren't using Blender, but one reason that truly makes sense is it's too risky to change their existing pipeline away from (typically) Maya. Which leaves new professional studios just getting started, who are beginning with a Blender pipeline. Such studios, whose future is hitched to Blender, are the best sources of long term funding. If you want to attract and retain these studios, you need to fix showstoppers, and IMO libraries and overrides are showstoppers.
1 Issue: Overriding Hierarchies.
Importance: Major Problem: You can't override a linked collection hierarchy. Description: @eobet: "The only way to link a hierarchy into your file is to link the collection its in. However, there’s no easy way to then override the hierarchy and make changes to it, since you can’t just make the collection local. You have to move the entire hierarchy to a new collection (which actually copies the links instead) and then you can unlink the first collection and start to work in the new one." blenderartists.org/t/links-library-overrides-are-still-glitchy-after-4-years-of-development/1476541 Commentary: This is a real pain. Often I bring a building into a scene and want to make a small change to it: Open a window, swing open a door, etc. This is a common use case... but there's no easy way to do this. You can go to create a modified version of the object, collection, whatever (hereafter "asset/etc") in the library for that specific scene, but that (storing scene specific customizations) is not what libraries are for! You can reconfigure the asset/etc so anything you might want to change is pushed up to the top, but that obfuscates the hierarchy, and isn't always possible (what if the building is embedded in an urban scene?). The ugly workaround I use is use Python to hide the object within the library asset/etc I want to modify, and then somehow recreate that object as an object in its own right (often nontrivial - because Blender doesn't provide an easy way to even tell the path of the overridden object, because you need to get not just what you're replacing but it's exact position and orientation too.) It ultimately works, but it's slow and messy, and I live with the fear a new release in Blender will come out where we can no longer do that. A similar thing happened in a 2.* release where suddenly you could no longer parent overrides: Ouch!, though that ability was later re-added. Suggestion: Allow the hierarchy of the linked instance to be opened up in the outliner (currently you can't - as @eobet describes), and on each object allow overrides to be created either (i) generally, or (ii) by path - which lets you have different overrides for different paths. I implemented this solution several years ago on an in-house 3D DCC and it works well. (No IP issues).
2 Issue: Identifying Linked Objects
Problem: Can't identify object in hierarchy. Description: Whether it's for #1 or just tweaking your library, often you see something on the screen, and you need to know what object you're looking at. There's no easy way to do this. I find it often takes about half-an-hour in a large scene to track something down. This is a real time sink. Commentary: The otherwise essential 'Edit Linked Library' Addon is of limited use here because it only identifies the library, but not the object you're clicking on, nor the hierarchy leading to it. I wrote a python script to identify the object, but I found no way to provide the hierarchy leading to it. Suggestion: When clicking on an object (which is a linked library asset), it should display/give access to not only identify the library, but the hierarchy all the way to the clicked object.
3 Issue: Broken Links, Resynching.
Problem: Broken Links, Resynching: Crashes, Bugs. Description: See blenderartists.org/t/links-library-overrides-are-still-glitchy-after-4-years-of-development Commentary: I find the same thing. Broken links are very hard to track down. The error message is vague (and the problem usually isn't visible in the outliner anyway - despite what the message says), so the only way to track the problem down is copy is to go through the console (which isn't easy to search), so I copy/paste to NOTEPAD, and look for 'Missing'. Resynching has issues: I find it crashes maybe 1 time out of 6 (which seems okay until you realize it needs to work 6 out 6 times!) Suggestion: More testing.
4 Issue: Even minor changes break libraries.
Problem: Even minor changes break libraries. Commentary: I've found lots of bugs with modifying an existing library. For example yesterday, I added just one mesh to a library. It should have made no real difference, but other objects in the scene - seemingly completely unrelated - just vanished. When I re-ran it, different objects vanished! It took five hours to recover the scene. Makes me so nervous about altering libraries that I have to version them and only change them as a last resort, which negates the advantages of using libraries. Even when it doesn't result to crashes, it frequently causes resynch problems. But the advantage of libraries is supposed to be that I can change something in one place, and that change will flow through to all scenes. This negates that advantage, and means once you create a scene with a certain library file, you can't change it. Suggestion: More testing. More robust resycning that can accommodate changes, perhaps use GUIDs to handle assets/etc which have been moved in the hierarchy or renamed (since Blender often renames/renumbers without telling you anyway.)
5 Issue: 64 character name limit.
Problem: 64 character name limit. Commentary: Productions using libraries will inherently be dealing with many more objects than those doing self-contained single scenes. We need the ability to label assets (anything you might access through a library - objects, rigs, materials, actions, etc) so they can be clearly and easily distinguished. For example, to signify that an asset is intended to be attached to a particular other asset, or that an Action is intended for a certain rig. This is particularly important given Blender's flat naming hierarchy. The 64 character limit really gets in the way here. I do get why Blender originally imposed this limit, but surely today's PCs have enough memory to allow longer names. (Anything up to 256). Suggestion: Allow longer names.
6 Issue: Can't override a lot of things.
Problem: Can't override a lot of things. Description: See blenderartists.org/t/links-library-overrides-are-still-glitchy-after-4-years-of-development/1476541 Commentary: Materials, and shape keys are often raised. There are other weird things too. I found with a library cloth object, I couldn't override the length of the animation, which of course will be different in just about every scene. Shape keys are particularly important though, because you need to be able to create corrective shape keys to certain rig poses in a scene. Having to put those in the library for scene-specific poses isn't just hard, it just doesn't make sense. Suggestion: Allow shape keys to be overriden. A few posters said Geometry Nodes will ultimately fix all this, but that is surely a long off, and shape keys are so widely used in CG (Blendshapes in Maya, Morph Targets in ZBrush) they're worth supporting. I'll leave others to comment on materials (since I haven't tried that for a long time).
7 Issue: Overrides lost on save.
Problem: Overrides lost on save Description: You can save .blend files, and load them in later. But often overrides are discarded!!! Ouch! This has been a problem all the way back to 2.*. Suggestion: What you save should be what you get when you reload a file. This is a very important principle for any application!
8 Issue: Granularity.
Problem: 'Make it local' makes too much local. Description: The usual workaround for override limitations is to 'make it local', but this loses all the advantages of overrides, including the ability to resync. Suggestion: Finer granularity. It would be better if we could override on a finer scale. Instead of having to 'make local' every piece of data in a data block, only override the actual piece of data we want to change.
9 Issue: Many addons incompatible.
Problem: Many addons incompatible with linked-library objects. Description: This is likely because linked-libraries are not widely used, so addon authors don't feel a need to support them, or aren't even aware of their existence, or that they need to write their addons differently. This is an obstacle to the acceptance of linked-libraries in the future. I've noticed many retargeting addons can't work with linked-libraries. Neither could the old version of jiggle. Commentary: One problem is those addons switching to EDIT mode, which is forbidden for linked-libraries. But why? Another is that every Property must be tagged with options={'LIBRARY_EDITABLE'}, override={'LIBRARY_OVERRIDABLE', and sometimes 'USE_INSERTION'}), or it won't be changeable. Wouldn't it be better to make this the default? After all, if I might want to change something in Append mode, why wouldn't I want to change it in Linked mode? Also last time I checked these weren't in the manual. Suggestion: Make these the default.
10 Issue: Override UX is unclear.
Problem: Override UX is unclear. Description: See blenderartists.org/t/links-library-overrides-are-still-glitchy-after-4-years-of-development/1476541 Commentary: Seemingly identical UX items do different things based on where you go into the menu. It's very confusing, the documentation isn't in step, and changes. e.g. in 3.6, suddenly you can no longer create an override from the outliner right-click menu. Need to go to the property tab and do it there (shift-click on link). It's weird that the outliner, when it works, lets you choose between Make Selected, Content, Selected & Content (still don't understand what that actually means - yes, I RTFM), but no where else does. Same with all the shift-clicking which isn't clear either. Much confusion. Suggestion: The Blender manual isn't terribly useful because it is often out of date and doesn't give the necessary detail. Stackoverflow and Reddit are more useful, but that's thanks to the efforts of those who figure it out through trial and error. A development process change could help though: Something superb Autodesk did was create itemized workflow lists in their documentation, which (i) made it very clear what you had to do to get a desired result, and (ii) showed that someone else had got it working, so you know you're not the first one to try it. (see thread on 2.8 cloth sim). It can be an itemized list with screenshots, or even a short YouTube video. Also, UX needs to be cleared up. I use overrides daily, and can mostly get them to perform, but 80% of the override menus are a mystery to me. Quote: @Hadriscus: 'I also don’t understand why the “make override” operators differ so much between the outliner, the object menu and the operator search.'
11 Issue: Ability to override local (non-library) instances.
Problem: Ability to override local (non-library) instances. Description: See blender.community/c/rightclickselect/5LZA Commentary: This would be useful too. See #1.
12 Issue: Transform Reset.
Problem: "Enabling a library override shouldn't reset transform" Description: See blender.community/c/rightclickselect/4Lrg/ Commentary: This is a simple thing, but it is such a pain!
13 Issue: Fixed?
This one ( blender.community/c/rightclickselect/Zzr6/ ) seems to be fixed?
14 Issue: Crashes and Bugs
Problem: BF may not be aware how buggy / crashy blender is. I kid you not: 30 crashes a day isn't unusual for me. (See 'Internet arguments' below). Description: IME it depends on what you are doing. If you are using a high-end PC for polygon modeling, sculpting, bugs/crashes are rare. But if you're doing the sort of things larger animation projects do, overrides, rendering, animation, you'll encounter bugs/crashes galore. I've seen Internet arguments on this - blaming the bug reporter of inexperience or their hardware of being non-standard, but having worked in IT, the bugs/crashes I see are symptomatic to untested execution paths. Basically, people attempting larger animation projects are doing alpha testing without anyone, except eventually them, realizing it. Unfortunately, their financial viability is hanging on this. This may be the reason Blender isn't being used by more professional studios. Commentary: Unfortunately reporting bugs in Blender is not easy. First it has to be repeatable, and many of the crashes aren't. All you can really do is reload and re-run and this works in most cases. On top of that, if you're going to do a proper bug report, it's a lot of work. You're expected to isolate the bug (on a large scene, days of work), stripping out gigabytes of assets, then test it on the pre-release versions. You also have to consider confidential assets in your test, which it might not be viable to remove. Also, .blend files are very leaky privacy-wise (Web search "Blend Anonymizer"). Plus IME bugs can sit around for literally months without being acted on, if they're ever acted upon. When I did a project on 2.8, I found an already-reported in the modifier for MDD which was years old, but had been "closed due to inactivity." (Desperate messages by the reporter felt like the Count of Monte Cristo!) Other times I've made reports which the blender devs haven't been able to replicate. Given that, (in the short term) it's better for all involved not to make the bug report and find a workaround instead. (Not good for long-term quality, but neither was closing that MDD bug report!) For another example, take crashes. If you look in the Blender bug repository, there are not many crash reports. Now do a web search for blender and crashing in the wider-web, and you'll find lots of them. The basic principle of Quality is that we catch defects as early in the production process as possible: The classic car-case is it's far cheaper to replace a defective screw before it goes into the car, instead of doing a nationwide recall later. But I do understand Blender's budget is limited, and of new features, testing, bug fixing, something has to give. But it would be good to at least gauge how widespread this is. Suggestion: On Windows at least, Blender.exe is launched through blender-launcher.exe. How about when Blender crashes, it logs it so it can be reported to BF statistically? (And allowed deferred reporting for when offline). Not the whole file due to privacy/confidentiality issues, but the fact a crash did take place (which the launcher can detect from a return code), and a stack trace so BF at least knows where the error occurred. This would let BF gauge the problem, because you can bet any established studio which pilots Blender, and encounters crashes, isn't going to report them. It just won't use Blender at all. (I remember a BF post complaining VFX companies hadn't contacted them. This might be why.)
15 Issue: Versioning
Problem: Not really a problem, because it's easily solved, but worth mentioning because it shouldn't be broken in future releases. When you have a large production, a corrupted library file could sink your entire production if you don't have a backup. Solution: Thus it's necessary to save versions of your library files. I solve that by adding a version number (e.g. 'foobar v10f.blend'). Unfortunately the Asset Browser doesn't comprehend this, so lists everything, which means it indexes old versions when most of the time you'd simply want to link to the latest version. I solved that my making a permanent library directory (permalib) and symbollicaly linking a versionless name of the file there (so 'foobar v10f.blend' would be linked as 'foobar PERMALIB.blend'). And if you need to use to an older version insteads of the latest, you can link directly (e.g. to 'foobar v09f.blend', in its own directly, not permalib) instead. That might be the case if you rerig an asset, but don't want to break older scenes animated on the old version. Commentary: This works as-is. Just mentioning it, because I'd hate to see a change which breaks it in a later release.
16 Issue: Asset Browser
Importance: Minor Suggestion: Allow keyword search (currently you have to can only search by exact substring, which makes finding things harder). When dragging asset into scene, place sensibly e.g. on dragged object, or ground place (currently puts asset at weird seemingly random positions you always need to correct).
I think Blender has won "The 3D Wars", for now. But libraries and overrides remain a troublesome province.
I hope we can improve libraries and overrides, because fix the above, and Blender would provide a very workable solution for large animation projects.
12 Comments
Well that was a long read. But a good one!
i didn't know how buggy and slow the library override is until i had to use it for a huge project... i hope it gets some attention from the developers soon
man if you could just explain it in a video and DM me i would love to create an explanatory video on it and re route the link here
Moving a .blend file to another folder losing all the links but when you just save from Blender inside that new folder everything remains intact... Never made sense to me and don't know if it's related but could be added to this list.
ah yes, also look at the changes in 4.0 related to library overrides. I don't understand, but from a superficial reading, some of the problems perhaps have been solved wiki.blender.org/wiki/Reference/Release_Notes/4.0/Core . The fixes were also in 3.6 wiki.blender.org/wiki/Reference/Release_Notes/3.6/Core
I don't think RCS is a good place to discuss such complex and important issues. It is probably better to contact Ton personally, or another managing person of the fund. It is advisable to do this together with other people who have experienced difficulties due to bugs in library overrides, and provide evidence why this area needs a lot of care
ahh writing a huge wall of text isn't very helpful. Just describe the feature you want to see implemented and add a diagram/screenshot to show how it works.
Also "PLEASE FIX BUGGY SOFTWARE" isn't a feature request, which is what 99% of your post boils down to.
Disagree, Omer. By Blender's fairly strict definition of bugs, most of these are not bugs (Just 14 - which is about crash handling, and parts of 3/4). Yes, walls of text are best avoided, but I'd hate for that vagueness to lead to a fix that doesn't address the problem. See OP where vagueness about 'versioning' caused confusion and turned out not to be a problem at all.
Looking back at it yes, 99% of the post doesn't boil down to fix bugs, it is more around 30%. 99% of the post however does not constitute actual feature requests imo. You spend a handful of paragraphs talking about how important Library overrides are, and one or too speculating on why studios do or don't use blender.
All this extra noise makes it hard to approach the post, wouldn't you agree?
Omer, I saw your posts on BA. I think you're trolling. I'm not going to respond any further.
I mean, to be honest, it is a fact that having one simple efficient post for each feature would have statistically more chances to reach a large public within the community and the devs.
Even if you just think of if from the "walls of texts are scary" point of view.
Having one post per feature also means you can go deeper in details and mockups on how to implement things in Blender. One example:
Display catalogs in the asset browser's main window — Right-Click Select
I put lots of details in that proposal that have just one feature, even though I could have just posted one big post with all the different things I would love in the Asset Browser. And it's already quite long as is.
But the other good thing is, the community can react and discuss efficiently about it. Whereas here we would need to constantly quote specifically what we talk about in each answer, scroll back and forth to search them up and discuss, ... And votes are meaningless.
@-L0Lock-, yeah, you can go both ways. But these are closely related so thought they were best considered as a group. Even things like the 64 character limit have a very big effect when you're naming assets for libraries. Also on the "wall of text", as Ross Perot once said: "I'm not going to soundbite that for you." :-) I used to write requirements for a living. Better to be specific rather than vague, otherwise you can end up being handed something you don't want.
Anyway, possible alternative to library overrides using Geometry Nodes: blender.community/c/rightclickselect/OZ6R