360º video camera tracking
I think it would be really cool to have a tracking option for 360º camera footage; I would think (but I could be totally wrong) that the current tracking algorithm could just be copied over with some relatively straightforward trigonometric adjustments.
2 applications of this:
-Simply taking your cool 360 videos to another level by incorporating VFX into them
-If you attach your 360 camera to the normal one you might be shooting a movie with, you could track the 360 footage rather than the normal footage and the algorithm would have so much more spacial information to work with! (Kind of a hacky application, but definitely the one that I'd be taking advantage of all the time!)
4 Comments
I don't claim to know all about this, but I don't think it's quite that simple. It depends on what you mean by 360 footage. The camera tracker, I think, uses lens correction in it's math. But 360 cameras at at least 2 lenses, often quite a few more. So my guess would be that you would have to also tell the software how many lenses, and in what configuration they are in. You'd also probably get better results if you have access to the video footage Prior to the hardware/software stitching. Because then it's just a regular VERY wide angle lens. But it Would be nice if it helped you with tracking the same feature across multiple cameras. So really, you can do this now....but you'd want it to be more automatic, and so would anyone else wanting to do this. You just either need the raw lens footage, or a method to undistort it back into that if you don't have access. I have Literally the cheapest 360 camera on the market, and it has the option to record both lenses at once without stitching though (only if I use the app to start/stop recordings though), so I'm guessing it's unlikely to not be an option?
If you have a really nice setup with one of those 12 camera goPros or something, then you Probably have enough overlap to create some really good scene tracking since every shot Always has 2 cameras shooting the same thing in a fixed relation to each other. There might be able to be some optimization there, since the camera's lens positions will always be the same, relative to the others. So that would probably require a new method. Although, if there's already a stereo tracking method then that might only require a small amount of tweaking? I haven't seen that though, but Also could be a good idea.
Thanks for the input, I totally forgot about the stitching! You're right, it probably is more complicated than I was thinking (though hopefully not so much so that the idea belongs in the waste bin lol)
No, not the trash. I was just pointing out that it's actually pretty complicated because it wouldn't use the same methods unless it can be used as 2 (or more) cameras separately.
I guess it is already possible in meshroom.org/ open source photogrammetry / camera tracking tool - you can turn 360 degree photos into fisheye projection and then use that data for camera tracking i think.