Blackmagic Forum

Hi,

Is there a way to get more detailed/defined output from the Depth Map in Davinci Resolve? If I compare them with other depth maps, like Midas or ZoeDepth, then Davinci Resolve's Depth Maps are somewhat barebones.
I know this is probably done for fast rendering, as they are done on the fly. But it would still be nice to have the option to decide for yourself if you want fast or detailed, or balanced.
As I want to convert 2D videos to stereoscopic 3D, a more detailed depthmap would help a lot more with the definition and probably somewhat less warping.
Would also be nice to be able to decide how the image/video gets extruded on the imageplane. Now I don't have any control over it, and the more I try to extrude, the wavier the sides are getting, giving a very ackward distorted view along the edges.

Cheers.

If you're looking for a more detailed depth map then you would need a higher controlled AI extraction. You may have luck running each frame through a program called Comfyui. It's a program for making Stable Diffusion set ups that allow you to create plates from the separate processes. Once you run each plate through, you could then see how they turn out, yeah them, and run some edge blending using the same technique you might use for a light wrap.

this might help you. https://vfxstudy.com/tutorials/depthmap/

I've had some success with Switchlight. I haven't touched it in months. They are developing it pretty fast and it's probably better than when I made this demo.

Ernest Skocdopole wrote:this might help you. https://vfxstudy.com/tutorials/depthmap/

Yeah, that one I already followed, to familiarize myself with the Depthmap tool. But it was not what I meant.

Igor Riđanović wrote:I've had some success with Switchlight. I haven't touched it in months. They are developing it pretty fast and it's probably better than when I made this demo.

Switchlight seems to be another subscription based tool. I stay clear of such things. So then I would use Relight. But I don't know if you can blend a normal map with a depthmap and if it would even add details to it.

bentheanimator wrote:If you're looking for a more detailed depth map then you would need a higher controlled AI extraction. You may have luck running each frame through a program called Comfyui. It's a program for making Stable Diffusion set ups that allow you to create plates from the separate processes. Once you run each plate through, you could then see how they turn out, yeah them, and run some edge blending using the same technique you might use for a light wrap.

I already installed Stable Diffusion locally on my computer and added the Depthmap generator (https://github.com/thygate/stable-diffusion-webui-depthmap-script) and tested out several of them, to find a balance between speed and detail. Although the default: res101 + BOOST was highly detailed, it takes a very very long time. Also tried ZoeDepth, that seemed to take forever.
Midas V3.1 gave me the results I need. "dpt_beit_L_512" seems to give issues with me, not able to generate the output to a file), but "dpt_beit_L_384" give me a satisfying amount of detail, while maintaining a decent enough speed. Although over an entire movie, it would still take a few days. Problem I faced with it was that it can't handle anything past a 3 minute video. Each frame it adds up to the RAM usage and Virtual Memory. They seem to first process each and every frame, while not generating it to a file. So even though it can process a 5 minute video, it isn't able to safe it.
I've already requested a feature on their page, to make the pipeline like in video-editors.

After your message, I've looked up a bit about this ComfyUI. Would it circumvent these kind of limitations from certain modules in Stable Diffusion?
I'm currently rendering the Deptmap node to a Saver/Loader in Davinci Resolve (estimated on 2 days, halfway there currently), as a temporary solution. But it would be ideal if I somehow could use Midas v3.1 as a Deptmap generator in Davinci Resolve, as that doesn't eat up all my RAM usage.

Cheers

Wow! You really did go down the rabbit hole! I'd say your just going to have to split the video into chunks. I don't think you're going to get any differences between the transition points from one sequence to the next. Just blend them together of you need to. I've only used comfy but you can use StableXL for higher detail but I'm not sure about the control net output saving out with the XL. It should work but I haven't done it.

You can also try using Nuke. They have a Diffusion system that you can get a little more detailed with and allows different Checkpoints to be used. They call the set up Copy Cat and the checkpoints Catteries. It won't be faster than what you're doing now, but it might be more accurate.

bentheanimator wrote:Wow! You really did go down the rabbit hole!

Is that something positive or negative?

bentheanimator wrote:I'd say your just going to have to split the video into chunks. I don't think you're going to get any differences between the transition points from one sequence to the next. Just blend them together of you need to. I've only used comfy but you can use StableXL for higher detail but I'm not sure about the control net output saving out with the XL. It should work but I haven't done it.

You can also try using Nuke. They have a Diffusion system that you can get a little more detailed with and allows different Checkpoints to be used. They call the set up Copy Cat and the checkpoints Catteries. It won't be faster than what you're doing now, but it might be more accurate.

Yeah, after your advice, I did cut them in 3 minutes pieces. Isn't there a way in Davinci Resolve to do that semi-automatically? Now, I had to input "+30000", do a cut (ctrl+< as I'm on Azerty) each time, making sure I didn't mess up.
I switched from Midas to Depth Anything on recommendation from someone on AVS-forum in Stable Diffusion (once I got Depth Anything working, as it was some 'special case'). And I got to say, night and day differenct!! It has the right details, much more than Davinci Resolves depthmap, and it doesn't flicker as much as Midas-model did. It still has some issues here and there, but it's an improvement on all the lines.
With Depth Anything though, I have to cut the movie into 2 minute pieces instead of 3 minutes (and manually generate each file seperatly).

But now I'm facing another problem. Took me a few days to figure out where the problem was occuring. As Stable Diffusion Depth creates an AVI which isn't able to be imported into any video-editor. So I convert them to MP4 with Handbrake. Then I import them into Davinci Resolve to stitch together and render.
Lining it out with the original video, I noticed a desynch between the movie and deptmap. Every now and then, there was apparently a frame missing.
First I thought it was the AVI file. I opened it up with Losslesscut, as it was the only app I seem to be able to open the AVI to check framecount. I found out the missing frames where in there. So the next thing I thought was that something went wrong with the conversion from AVI to MP4. Even if I stitch the parts I have with losslesscut and converted them to MP4, it had random frames missing, this time different ones than with the 2 minute parts.
So I opened up the MP4 in LosslessCut. And yet, all the frames are there. So something is going wrong in Davinci Resolve, that it loses frames when importing the file. But I don't know what. Is there a setting I'm missing?


Cheers.

I personally like rabbit holes, so I meant it as a compliment.

I'm glad SD is working for you. Usually in these kinds of instances, I tend to use image sequences. There are too many places where you can put in the wrong fps or starting frame when transcoding. If you want to know which frames are missing, you can add you new depth videos onto a layer above your original and switch the color mode to Difference. Then when there is a miss-sync it should be obvious which frame it is.
But before that, I would try using image sequences instead of transcoding to AVIs and MP4s.

Indeed, with putting it atop the original movie, I could see where the frames were missing. Although I lowered the opacity. But I'll try that color mode Difference next time.

Now, those missing frames appeared to be from converting AVI to MP4 with Handbrake. No matter the setting, it would always skip a frame here and there. Funnily enough, the Windows Movie Maker Video Convertor did do a good conversion. So missing frames weren't an issue anymore.
Although now, with my 3rd movie, all of a sudden I had Depthmaps that sometimes added a last extra frame, so instead of merging the files together with a tool like LosslessCut, I had to import every slice of the movie into Davinci Resolve, on top of the original movie and do the same thing as with the missing frames, but now removing frames. Luckily that is easier to achieve, but still again, extra work.

Don't know why it sometimes adds an extra frame. This time I made sure my movie was cut on cuts of a shot, so to not create discrepancies in color in the Depthmap from 1 slice to another.

Don't know if you can do Image sequences with SD. And I don't know, for an entire movie, it feels like too convoluted, because we're talkint about around 160.000 frames and more.

Cheers.

If you are using Handbrake you could try switching to Shutter Encoder, that will give you all the options you should ever need.

https://www.shutterencoder.com/en/