Replies: 11 comments 21 replies
-
|
also you can try it yourself in their Colab |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I'm aware of the paper, I quote their observations in the readme. Your results are pretty amazing, I was planning to test their method in the extension, but did not notice LeRes was so much more detailed. As a script this would have made installing too cumbersome, as an extension it is now possible to easy install more dependencies. I will look into it. |
Beta Was this translation helpful? Give feedback.
-
|
Some conversation about LeRes model implementation in to the depthmap2mask extension. It might be interesting: |
Beta Was this translation helpful? Give feedback.
-
|
I'm working on it, the imports are a problem. I have the res101 net working now, just need to do some testing before i push it to the repo.. edit: it's live. |
Beta Was this translation helpful? Give feedback.
-
|
This is wonderful news! A big thank you for that research and for pushing your extension to the next level. I'm really glad of you and your work! |
Beta Was this translation helpful? Give feedback.
-
|
I also want to personally thank you for making this happen. |
Beta Was this translation helpful? Give feedback.
-
|
You're very welcome. |
Beta Was this translation helpful? Give feedback.
-
|
You're very welcome, this is not yet multi-resolution merging as implemented by BoostingMonocularDepth, I will look into that next. The intention/scope of this script was always to get the best possible depthmap and 3D related formats. This includes multi-resolution merging by BoostingMonocularDepth. I initially wasn't planning on including LeReS until I saw the results above. Mesh and point-cloud generation are also future goals, including vt-vl-lab/3d-photo-inpainting. I could make those separate tabs together with BoostYourOwnDepth .. i'm not sure yet. There's also a great fork by @graemeniedermayer stable-diffusion-webui-normalmap-script focusing on normal maps with some very nice results already. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks a lot for pointing out that normal map script - I had completely missed it and I was in fact looking into exactly that kind of thing just yesterday to produce some low-relief stone carvings. |
Beta Was this translation helpful? Give feedback.
-
|
Something super cool I recently found with lighting is you can use img2img to rebake in the new lighting from normal maps (this might work for depth maps too) and it seems to remove a good portion of the lighting artefacts. I don't mean to be self promo-y but here's my quick explainer. I also did another mini-project using the projected depth maps in mobile augment reality and they look really cool! Thank you thygate, this has opened up a lot of possibilities! |
Beta Was this translation helpful? Give feedback.
-
About your explanation, Wow, I didn't realize how deep we can get it. Thanks for inspire! |
Beta Was this translation helpful? Give feedback.
-
|
In my testing so far, res101 seems to generate more detail compared to dpt_large, but sometimes gets very wrong depth estimation for some objects, and doesn't always seem to recognize windows .. It still suffers from the same issue as the midas models, there is a trade-off between structural consistency and high-frequency details with respect to net size (see observations). Multi-resolution merging is the way forward. What are your experiences and thoughts, is LeReS really better overall ? |
Beta Was this translation helpful? Give feedback.
-
|
That actually sounds excellent for normal maps! |
Beta Was this translation helpful? Give feedback.
-
|
I am not satisfied with any of them. Yesterday I had to find a way to transform a 2d digital painting of a tree into a 3d model, and basically I used all the different algorithms I had access to to generate depthmaps - so I had 5 of them - and then I manually massaged them in photoshop to get what I wanted out of them before using them as source for displacement maps. What also helped a lot was that the digital painting I was working from was split in layers so I could work on each element separately and then reassemble them in 3d space. But it is my understanding as a user (so I might be completely wrong as I have no grasp on the programming side) that indeed multi-resolution merging is the way forward. Is LeRes better overall ? It is clearly more detailed, so if you are more into extracting relief information from a surface, like the bark from a tree, it is clearly better than anything else. For some reason, the one that seems to give me the most "realistic" results is the DPT-Hybrid algorithm. I wish I could tell why, and why it doesn't at all on some images. It reminds me a bit of the upscalers: some are better than others, but you'd be hard pressed to find the "absolute best" that would be ideal in all situations. And just like with upscalers, it might be best to use them in combination. What about that Structure-Guided-Ranking-Loss algo ? Is that being used anywhere ? Is it related to https://github.com/aim-uofa/AdelaiDepth that you implemented recently ? One more question while I'm at it: is there any special advantage in the use of heatmaps instead of deptmaps ? Do they allow to encode more than 8 bit of depth information somehow ? I feel like this could be the case by not have all the RGB channels being always equal to each other, but I'm just guessing. Never used them before, except when I had to work with actual thermal cam footage, and even then we would set it to output greyscale instead. |
Beta Was this translation helpful? Give feedback.
-
|
One more thing: donlinglok pointed out something that might be useful for merging LeRes and Midas: vt-vl-lab/3d-photo-inpainting#136 You might want to have a look at that. |
Beta Was this translation helpful? Give feedback.
-
The people at BoostingMonocularDepth seem to be least excited about SGR, and it might be a pain to implement in the script, I have no intentions of adding it at the moment.
To make the depthmap easier to analyze for human eyes, Show HeatMap shows an extra image in the WebUI that has a color gradient applied. It is not saved. edit: messed up reply again. |
Beta Was this translation helpful? Give feedback.
-
|
I tested v0.2.3 and cannot reproduce my previous results. I made img2img with denoising strength = 0 to remove the influence of the generative model, plus the DepthMap v0.2.3 script with res101 as the model. I adjusted the image size to match the original. Perhaps I used incorrect settings in this test, but I'm hoping someone can correct me or run the same test with their settings. |
Beta Was this translation helpful? Give feedback.
-
|
It's the next thing I will be looking at .. |
Beta Was this translation helpful? Give feedback.
-
|
Did you mean that? They call it MergeNet |
Beta Was this translation helpful? Give feedback.
-
|
That's the stand-alone implementation of their merging operator, it allows for you to use only 2 depthmaps of different resolution that you provide and merge those. I might add support for this later in a separate tab or something, i'm not sure. I'm counting on the algorithm explained in BoostingMonocularDepth to select the regions and net sizes to generate the depthmaps at different resolutions to feed to the merging network to be the best possible output. |
Beta Was this translation helpful? Give feedback.
-
|
I hope this algorithm does not take too much of your time. Thanks for the conversation, and a big thank you for your time! |
Beta Was this translation helpful? Give feedback.
-
|
You're welcome. I'm taking a slight detour looking at the 3D reconstruction code in the LeReS repo. It can output a PLY point-cloud, and the results here look pretty good. |
Beta Was this translation helpful? Give feedback.
-
|
I have the multi-resolution merging as implemented by BoostingMonocularDepth working, above was generated in the WebUI using leres and the image from bugmelone's start post. Still have some things to clean up and test, but it's coming soon.. |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
it's live. |
Beta Was this translation helpful? Give feedback.
-
|
Multi-resolution merging is now implemented and significantly improves results, and res101 can now also compute on CPU in the new version. Let's see some results =) |
Beta Was this translation helpful? Give feedback.
-
|
This is insane! It's works super well on everything I've thrown at it so far! new even the much more ambiguous scenes it does a great job at new |
Beta Was this translation helpful? Give feedback.
-
|
How about a 3D stereo/anaglyph image .. =) Your picture looks pretty amazing, check here : #45
|
Beta Was this translation helpful? Give feedback.
-
|
Wow, this looks really cool, it's time to watch it on the big screen! Thanks a lot!!! |
Beta Was this translation helpful? Give feedback.
-
|
ah yes, i forgot adding an explanation to the readme for the setting. The default value is 3000 indeed, it will determine the max "wholeimage size" which sets the sizes for the intermediate depth maps it generates. I set it to 1600 default so people with little VRAM can run it too. And you're very welcome, the separate tab was the most requested feature for simpler processing of existing images. |
Beta Was this translation helpful? Give feedback.
-
Just wonder if it possibly to make?
Found this repo today:
https://github.com/compphoto/BoostingMonocularDepth
and i was surprised about level of details in output maps
Beta Was this translation helpful? Give feedback.
All reactions