3D Photography using Context-aware Layered Depth Inpainting

[sina-masoud-ansari]

Hello, just wondering if there is any interest in integrating this?

I managed to get it working and it might be a cool way to let people experience their creations in simulated 3D without any additional gear

So far it only seems to work properly with midas and makes a call out to boost. I've provided an example below

If there's interest I can create a fork of their repo with the changes needed to get it working (it had some OS specific commands and the requirements was out of date)

Integrating it is likely a matter of pointing to the right files assuming the dependencies play nice.

statue_circle.mp4

[thygate]

Sorry for the late reply, I saw it on mobile and meant to answer on desktop but got caught up in things ..
So, the 3D Inpainting paper has come up a number of times in different threads and is in the pipeline. I'm not in a hurry with this one, as the pointcloud ply still needs some consideration for viewing, and the video generation has potential, but needs some thinking about how to generalize the motion without getting into deforum turf etc .. And how to integrate it in the ui, layout wise, .. suggestions are welcome !

I wanted to look at cython or numba, maybe implement that stereo generation function in cython for a speedup because reading/writing single elements in python is painfully slow.

I also still wanted to add support for manual merging of maps using boost, and I wanted to try some things to get a threejs viewer into the webui ..

So plenty to do still, ..

Also wanted to investigate Parallax Mapping..
in this spiderman game all the skyscraper inside rooms seem modeled, but are apparently depth maps : https://www.youtube.com/watch?v=eX7x1mJrQJs
different implementation in garry's mod : https://www.youtube.com/watch?v=W9LJNxqP8dA

and then there's still these 3d reconstruction papers :
https://shunsukesaito.github.io/PIFuHD/
https://huggingface.co/spaces/Yuliang/ICON

edit:
hot off the plate :
https://github.com/openai/point-e

[sina-masoud-ansari]

No worries :)

Wow, yeah lots in the pipeline already. Parallax mapping would be neat but I guess that needs some specific shader / webgl integration.

The 3D reconstruction stuff is incredible, didn't know that was possible!

[sina-masoud-ansari]

Oh brilliant! Look forward to testing #51 out.

Yeah might make sense to use a separate extension. Or perhaps a library with these various tools since there's so many cool things happening in this space.

I've successfully extracted the whole stereo gen. process into a separate program so I can run SD and stereo gen. on different GPUs since it's quite time consuming to convert a batch of video frames and I wanted to use SD/VR in parallel.

So now I just need to connect the image->depthmap->stereo workflow to the video->frames extractor and frames->video process I wrote recently to get an end-to-end mono2stereo video pipeline.

The tricky part is the boost library integration and all the options there, so I'm looking at how to make that dependency cleaner. Maybe by making boost it's own library, or just extracting the necessary file dependencies. Not sure if you have any thoughts around that?

The bottleneck seems to be depthmap generation, so I think the next step is to batch the frame processing over multiple-gpus and then make it easy to deploy and run remotely, because anything at modern resolutions that's more than a few seconds is going to take ages.

[sina-masoud-ansari]

I think pulling out the boost repo files should be ok, there's 8 of them and then I can refactor the options handling and strip out other code not being used

[thygate]

I've done some short 4k clips overnight without boost, it's doable for short videos with the numba speedup. I was using ffmpeg to extract frames, audio and recombine them.

For reference :

extract audio:
.\ffmpeg.exe -i '.\vid.mp4' -q:a 0 -map a audio.mp3

extract frames :
.\ffmpeg.exe -i '.\vid.mp4' frames_in\$frame%05d.png

recombine processed frames :
.\ffmpeg.exe -r ORIG_FRAMERATE -i frames_out/%05d.png -i audio.mp3 -vcodec libx264 -crf 25 -pix_fmt yuv420p -acodec copy -shortest out.mp4

Splitting the depthmap generation into a separate pass and then running the stereo generation afterwards with compiled native code will probably still be somewhat faster. (the stereo image repo has a C++ implementation of those 2 functions) It currently runs a single thread, so lots of room for improvement there.

I cloned the needed repos, boost and midas (cause boost contains old midas) into /repositories so I could not edit them, so I had to pull everything into the script that couldn't find imports or needed editing to function.
I could have just copied it all into this repo and go from there, but the initial thought was to keep the cloned repos intact, so other users/extensions can use it there too.

If I would start over, I would probably copy everything locally and edit it ..

[sina-masoud-ansari]

Excellent thanks, yeah I was using boost so that was prob. overkill for video :) res101 without boost seems acceptable performance wise. I had some success with Moviepy for managing the calls to ffmpeg, the overall process looks like this in case it is useful at some point:

from functools import partial
from moviepy.editor import VideoFileClip

def as_stereo(model, pix2pixmodel, device, boost, image):
    # this calls out to the depthmap script functions, generating depth map (with or without boost)
    # and generating the stereo image
    return convert(image, model, pix2pixmodel, device, boost=boost)

device = torch.device("cuda")
boost = False

# loads a depth model e.g. res101 and merge model if boost requested
model, pix2pixmodel = load_depth_model(device, boost)

# defines function to be used with Moviepy per frame filtering method fl_image
fl_func = partial(as_stereo, model, pix2pixmodel, device, boost)

path = "mono.mp4"

with VideoFileClip(path) as video_clip:
    stereo_clip = video_clip.fl_image(fl_func)
    stereo_clip.write_videofile("stereo.mp4")

[thygate]

Thanks for that example, that certainly makes things quite a bit easier to implement =) I will definitely come back to this..

And yes, I agree, res101 performs fine with large enough net size for video.

We just had another contribution improving the stereo image generation technique, check it out, another significant improvement!

I will be implementing midas 3.1 next, solving the current issue with cloning an older commit of it, and bringing a few new depth models..

[AugmentedRealityCat]

Hello, just wondering if there is any interest in integrating this?

I've had loads of fun playing with this over the last two months. Here are a few links that may be of interest to you:

Version running on windows, adapted and tweaked by @donlinglok
https://github.com/donlinglok/3d-photo-inpainting

Discussion about using this with Automatic1111 WebUI
AUTOMATIC1111/stable-diffusion-webui#4268

[thygate]

The 3D photo inpainting is slooooooooow ..

[thygate]

No options yet, but next version has 3D photo inpainting, I need to do a little more testing and will release asap..

[AugmentedRealityCat]

If you need help with testing, let me know, it would be a pleasure to help you in any way I can.
But the 3d photo inpainting process is indeed slow, particularly the "now writing PLY and doing basically everything" part (I'm writing this from memory - the exact wording is probably different).

Now that you have played with the code, can you describe the difference between the 3d-photo-inpainting process and the blender depthmap input add-on ? How much "inpainting" is actually happening, and how ? Since the Blender add-on is easier to control, I've been using that to extract 3d models instead of 3d-photo-inpainting for some time now. But your next version might change all that !

Is there any way to re-use the code you have implemented for filling the gaps in stereo image pairs as inpainting color information for the 3d-photo-inpainting process? What is the upcoming version of your extension using as a 3d output format? Have you kept the basic PLY + vertex color map?

Thanks again ! Your extension + the depthmap input hack have been a revolution in the way I use Stable Diffusion.

[thygate]

I just made the commit, have at it =)

It was crashing my debugger a lot in the beginning before i realized the bilateral filter was not optional ..

I split the mesh generation and video generation into two sections .. So it's easy and quick to generate videos once you have the mesh.
The UI is a bit messy like this, not sure how to make the layout, suggestions welcome.

Initially the video generation required the rgb and depthmap and the mesh, but was able to get rid of it by writing the required value into the ply comments. It already stored original width, height, and fov like this. So now it only needs the mesh to generate videos.

I optimized the read_ply function a bit, got me a 20% increase in speed for (re-)reading the mesh.

As far as I understand it projects in the camera perspective, but segments the mesh into 'layers', whose edges are inpainted by some margin. There are still holes, but the overall freedom of movement before they become visible is increased compared to the method previously discussed / the blender addon.

You can import this mesh into blender, and use the 'Color Attribute' Node to plug into the Shader 'color' to get the mesh rendering in color, and set the camera to origin and pointing up to align them. Might also need to set the FOV to match.

It also occurred to me that this might also be a source for filling the gaps in the stereo images, but the current method is near realtime while this method requires a dozen minutes or so for a single frame ..

Yes, the 3D format is ASCII PLY, list of vertices with rgba color and list of faces. Converting to Wavefront OBJ should be easy, but is equally bulky due to the ASCII nature. Is there such a thing as .ply.zip, or .zply, I might consider implementing something similar as the files are quite large.

Results are quite impressive, I've generated some very pleasing slow long panning shots of that spaceship image that could go straight into a movie..

[thygate]

depthmap-0000-0001_.mp4

original image from #34 @bugmelone

[keenanamigos]

Hey all! Seems like I was tagged by accident here - just wanted to give a heads up so that correct person could be tagged 😄

[thygate]

I deleted some spam earlier today, lots of names were in it .. probably what it was ..

edit: just checked (still have the email notification), and yes your name @keenanamigos was in it .. So you can ignore the tag =)

[Thermotronica]

This is amazing

[Thermotronica]

Can i share it?

[thygate]

Did you mean to quote someone, what is it you wish to share ?

I'm pretty sure can you share anything in this thread ;)

[graemeniedermayer]

3d photo inpaint is actually insane! Great job on the integration. Even with very chaotic pictures it does a amazing job.

depthmap-0051_swing.mp4

[thygate]

That is a great showcase for it !

[AugmentedRealityCat]

Finally got some time to test this. Just did a first full run including the 3d model and the 4 videos - it took 3m52s for a 1024x512 image.

(EDIT: it took that long because it downloaded the extra models for 3d-photo-inpainting - I just ran the same thing again, and this time it only took 2m43s ! this is much faster than the standalone version !)

I got this warning along the way but everything seems to be working !
Should I worry about the warning ? Here is the log in case this information might be helpful.

WARNING:py.warnings:C:\stable-diffusion-webui-master\extensions/stable-diffusion-webui-depthmap-script/scripts\inpaint\mesh_tools.py:174: RuntimeWarning: divide by zero encountered in divide
  input_disp = 1./np.abs(input_depth)

Now, time to check that PLY model and see if the pipeline from SD-to-blender-to-C4d I was using with the standalone version of 3d-photo-inpainting is still working. Normally it should. EDIT: Everything works. The inpainting works - that was the main thing I was looking for ! Combined with the latest model (the 512 edition) I got amazing results. I am rendering a 30s slow-motion flyover in C4d - just a plain camera move. I'll post it later. In the meantime, here is one of the 4 automated ones:

depthmap-0266_circle.mp4

[donlinglok1]

wow!🤯

[AugmentedRealityCat]

The last frame is interesting because it shows some of the limits of that technique:

1. There is a tower on the middle right side with its top detached from the land - there is a strip of sky between it and the island.
2. There is a lot of texture stretching on nearly-horizontal surfaces.
3. The trees and many other elements are just 'painted' on the 3d surface, and are not much 3d themselves.
4. Some of the outpainting border is visible near the two largest towers at the top left - but it's much better than the first clip.

When it's moving though you hardly perceive any of those little oddities.

[SlimeVRX]

I want to reproduce the result like your video above. Are there any more detailed instructions? I want to adjust the camera movement

[benedlore]

overlay a 4k bitmap texture on the vertex color map

@AugmentedRealityCat
Could you describe how you accomplished this a little bit more? I don't use C4D, but do use Max, wondering about how you were able to go from vertex colors to UV space to use a 4K texture map instead of the included vertex colors in the output model. Also, how did you get it to generate a mesh in 2 minutes at that resolution? I have never seen anything close to that.

[AugmentedRealityCat]

Since the 3d model is extracted from a picture I simply project that picture back on the model as a texture map applied in camera-space. The vertex colors were kept as a layer underneath, but they are not as detailed and there are some areas showing slight color mismatch.
If you watch closely you can see that border between the projected bitmap and the vertex color in the blue of the sky near the two towers at the top, particularly towards the end of the clip.

It should be possible to do this with 3ds max, but I must admit it's been too long since I've used it to remember how. Basically it comes down to this:

background layer: per vertex colors, no UV required
foreground layer: 4k image, UV based on camera-space
layer mask: per pixel camera visibility

[thygate]

@graemeniedermayer I added your youtube video to the README, I hope you don't mind .. ?

I also linked to this thread, absolutely awesome video by @AugmentedRealityCat, I would love to add it too with your permission.

[AugmentedRealityCat]

Go ahead, I give you my permission to add my little video to the README.

Should I also share the prompt, parameters and photo I used to synthesize the base picture ?

Would you like me to write some kind of walk-through tutorial to get something similar to my first clip, the one with the inbuilt circular camera move ? Let me know if you are interested and I'll look into this, but probably only next week though.

Here is the pic I used for IMG2IMG - I cropped it to keep only the central rocky island (the picture is public domain on wikimedia):
https://upload.wikimedia.org/wikipedia/commons/3/39/Rocher_perc%C3%A9_.jpg

[AugmentedRealityCat]

More examples from production pipeline prototypes I've been working on:

model generation from drawing

Truck_drawing_v02.mp4

model split into 2 separate meshes + closing mesh holes + manual inpainting + animation test

Hamster_Toy_v17.mp4

[donlinglok1]

🤯the hamster head can move

[Absinthze]

Hi! Is there a way to export the mesh and texture into a 3D application?

[AugmentedRealityCat]

You definitely can. The depthmap extension has an option to output a PLY mesh with colors-per-vertex (RGB Vertex Map). This older format is not supported by all applications, but you can use Blender, which is free, to import it and export it to another more common format like FBX, OBJ or ABC.

That's how the drawing of a truck above was made: I imported the model in Cinema4d via Blender, and I simply animated the camera over there as it's more flexible than trying to define camera moves with formulas directly in the Depthmap Extension video-output module.

But there are many other ways. For the Hamster plush toy animation above, I worked differently and I did not use the PLY export function at all. Instead I worked directly in Cinema 4d and I applied some kind of displacement effect based on the depthmap as an input. Basically, this either push or pull each point on the grid according to the value of the depthmap at that point. That gave me a sofa with the hamster completely stuck with it - a bit like if both were covered by a fabric on which we would have printed the colors of the objects hidden underneath.

The next step was to manually separate those two extrusions as two separate objects. There are many ways to achieve this, but here I simply took a polygon selection tool and I selected the area covered by the Hamster and I split it from the rest. Now I had two separate models. I removed the extra polygons that were attaching the back of the Hamster toy with the sofa, and then I filled the holes behind the hamster and on the sofa.

That hole in the sofa though did not have color information - by default it would take the colors of the hamster in front of it. So I took that texture into Stable Diffusion and I inpainted the area covering the hole on the sofa so it would be seamless with the rest. And I took the result of that operation back into Cinema4d to use as a texture map on the sofa.

The last step was to adjust the geometry of the Hamster plush toy to make the back of it rounder, and to animate it very simply just to show it was indeed detached from the other object.

In the tool I imagine we'll be using in the future, most of these steps will be automated and we will get automatic 2d-to-3d scene extraction with object separation and inpainting of all hidden surfaces + automated labeling + everything ready to be animated by prompt. I wish I were a programmer !

[AugmentedRealityCat]

If you want to see something completely different, another 2d-to-3d AI pipeline I have prototyped is based on the DMT-Mesh text2mesh add-on for Blender, along with the Dream Texture add-on. This one doesn't use Automatic1111 at all - the Dream Texture add-on has its own diffusers based version of Stable Diffusion with hacks to use custom depthmaps as inputs, and the DMT-Mesh prompt-to-3d model functions are based on completely different code, with one algorithm dedicated to transform text or pictures into Point Clouds, and a second algorithm to transform that point cloud into a mesh.

Here is the link: Firework-Games-AI-Division/dmt-meshes#2 (comment)

(I prefer not to post the video itself over here in case people think it was made with the depthmap extension - I don't want to mislead anyone.)

I wish I could get someone to adapt this to work directly in Cinema4d as this would be much more convenient for me.

[AugmentedRealityCat]

Here is yet another pipeline test: this one shows the Image-to-PointCloud and then PointCloud-to-Mesh to synthesize a post-apocalyptic pickup truck, and Dream-Texture to create a ruined city environment for a quick animation test.
Firework-Games-AI-Division/dmt-meshes#7

[thygate]

Again very impressive @AugmentedRealityCat , I've been away for a while, will be catching up asap.

Is WaveFront OBJ output still desirable for the inpainted mesh ?

[AugmentedRealityCat]

Is WaveFront OBJ output still desirable for the inpainted mesh ?

OBJ is the obvious choice: it is supported by most 3d applications I've came across, and it's been there for a long time.

Other formats to consider would be FBX and ABC.
FBX allows embedding of texture maps directly in the file, something OBJ doesn't.
ABC is the most recent standard, and is better than OBJ and FBX in every way I can think of, but it's not as widely supported yet.

All these formats support per-vertex colors, which is how colors is encoded inside the PLY model when you use the 3d-photo-Inpainting code. It would be useful to have the option to convert the vertex map into a UV texture, but this is not as simple as it sounds as this would require both automatic UV unwrapping and vertex color baking.

[steampunkchris]

Yes!  OBJ or FBX would be preferred =)Sent from my iPhoneOn Jan 29, 2023, at 2:23 AM, Bob Thiry ***@***.***> wrote: Again very impressive @AugmentedRealityCat , I've been away for a while, will be catching up asap. Is WaveFront OBJ output still desirable for the inpainted mesh ? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[thygate]

@steampunkchris @AugmentedRealityCat
Wavefront obj (with vertex colors) is now supported, enable under settings.

[NameRX]

Hello!
Thanks for this amazing feature!

Could you please add an option for loading custom Z-depth images? Sometimes I edit generated z-depth and it would be great to load an edited png.

Also it would be nice if you add an ability to export into a png sequence or some other lossless format, even jpg at lowest compression would be great.

Thank you!

[AugmentedRealityCat]

i'm not a professional programmer, and I have never written python code before, so it may be imperfect and buggy, sorry about that. but it works :D mod.zip

I never managed to get your previous version to work but I wanted to check with you if you had kept on developing it. It would be a very useful tool.

[SlimeVRX]

I also look forward to this job

[NameRX]

i'm not a professional programmer, and I have never written python code before, so it may be imperfect and buggy, sorry about that. but it works :D mod.zip

I never managed to get your previous version to work but I wanted to check with you if you had kept on developing it. It would be a very useful tool.

Hi!

What system do you have? I run it on windows 10 with Python 3.10.6, DepthMap v0.3.8.

here is entire directory of my mod, please save original files, and try to replace with this one. Files are located in \stable-diffusion-webui\extensions\

stable-diffusion-webui-depthmap-script.zip

[SlimeVRX]

Thank you! I will try it

[AugmentedRealityCat]

What system do you have? I run it on windows 10 with Python 3.10.6, DepthMap v0.3.8.

Windows 10, python: 3.10.6 and DepthMap v0.3.8 as well. But maybe I was not a couple of weeks ago when I tried to run this. Thanks for sending the zip - I'll try that soon and I'll give you some feedback.

Have you seen the new ZoeDepth extension for Automatic1111 ? It's great really - you should try it if you haven't already, it might give you some ideas or solutions. The 3d panorama extraction is a really nice trick as well - I used to do that manually, but this is so much faster !

https://github.com/sanmeow/a1111-sd-zoe-depth

[NameRX]

Here are my results:

Mountains_AI_01_2.mp4

A still picture was generated in midjorney, then I changed it a little bit using img2img in SD, then I made several upscaled pictures using SD Upscale and combined them in photoshop with Topaz Gigapixel output to fix some bad areas. After that I fixed overstretched edges at last frame by adding some details using img2img, and after that reprojected fixed frame to whole video using Ebsyth. That made a clear good looking footage at all frames. I combined all outputs in After Effects and to make it look better I added some digital compositing like moving clouds and shadows from them. Final output is in 4K.

Easy!

[LanceGao97]

It says this under that excute line，
Traceback (most recent call last):
File "/content/3d-photo-inpainting/main.py", line 6, in
import vispy
ModuleNotFoundError: No module named 'vispy'

[LanceGao97]

I never managed to get the code shared by @NameRX to work when I tried it and I haven't tried again since. Everything I did with 3d-Photo-inpainting on Windows was with the help of @donlinglok - he is the programmer who knows about these things and who made it work for me before it was adapted for this extension. Now I use the extension to do those things instead.

OK thanks for reply , I already got the PLY files, I killed many hours finding ways to batch render them into videos. I m still struggleing, I can only do basic copy paste, I don't know any python .... This is a nice Wolrd. Struggle always.

[donlinglok1]

It says this under that excute line， Traceback (most recent call last): File "/content/3d-photo-inpainting/main.py", line 6, in import vispy ModuleNotFoundError: No module named 'vispy'

Seems the library is too old and need many upgrade to run, even the BoostingMonocularDepth didn't update for 2 year.
You may need to look for other example.

[LanceGao97]

It says this under that excute line， Traceback (most recent call last): File "/content/3d-photo-inpainting/main.py", line 6, in import vispy ModuleNotFoundError: No module named 'vispy'

Seems the library is too old and need many upgrade to run, even the BoostingMonocularDepth didn't update for 2 year. You may need to look for other example.

THank you anyway for reply, I think it s time to give up for me, I tried 4 on line server , all fail to render videos, to be wise, is just render it with my local 4070ti, and don't think about it anymore, other wise, I might got lost and frastrated. THanks again.

[charchit7]

Here are my results:

Mountains_AI_01_2.mp4
A still picture was generated in midjorney, then I changed it a little bit using img2img in SD, then I made several upscaled pictures using SD Upscale and combined them in photoshop with Topaz Gigapixel output to fix some bad areas. After that I fixed overstretched edges at last frame by adding some details using img2img, and after that reprojected fixed frame to whole video using Ebsyth. That made a clear good looking footage at all frames. I combined all outputs in After Effects and to make it look better I added some digital compositing like moving clouds and shadows from them. Final output is in 4K.

Easy!

Hey amazing work @NameRX could you please share your code to replicate it>