In Outliner, there should be options for invert filter selection with for example alt + click