Curved Depth Map

[marcomiglionico94]

Hi, I am trying to create a visualizer that allows me to visualize the room layout. I am using a custom image for inference.
The results look good, when i check the segmentation overlayed on my image. The main issue I have is when I try to use open 3d to plot, it looks like the depth map is curved. Do you have any explanation for this?

This is the overlayed image

This is my Open3d visualization of the polygons:

The main issue is the curvature, and that is due to the depth map extracted by the network. Any idea of how to solve it?
Thanks

[bertjiazheng]

Hi @marcomiglionico94

I think this may be due to the incorrect depth distance we used. You may try two different depths: i) distance from the point to the image plane or ii) distance from point to the camera center.

Best,
Jia

[marcomiglionico94]

Hi @marcomiglionico94

I think this may be due to the incorrect depth distance we used. You may try two different depths: i) distance from the point to the image plane or ii) distance from point to the camera center.

Best, Jia

Hi thanks for the quick answer, where exactly I can find that distance parameter that you used? Can you refer to the file or line of code where I can find it

[CYang0515]

For a point (x,y,z) in 3D space, the depth map extracted by network is the z coordinate, not the np.sqrt(x^2+y^2+z^2).

[marcomiglionico94]

Hi @marcomiglionico94

I think this may be due to the incorrect depth distance we used. You may try two different depths: i) distance from the point to the image plane or ii) distance from point to the camera center.

Best, Jia

Is it possible to know at least what kind of visualization you use for the paper? Did you use Open3D or something else?

[marcomiglionico94]

For a point (x,y,z) in 3D space, the depth map extracted by network is the z coordinate, not the np.sqrt(x^2+y^2+z^2).

I know that, but in the code the depth map is basically created in ConvertLayout function, in particular in this part of code:

n_d = param[:3] / np.clip(param[3], 1e-8, 1e8) # meter n/d n_d = n_d[np.newaxis, np.newaxis, :] inverdepth = -1 * np.sum(np.dot(n_d, K_inv) * ixy1map, axis=2) depth[mask] = inverdepth[mask]

My question is which parameters can I change so that the depthmap is not curved? I imagine that in the reconstruction you show in the paper you used a depth map, to make the 3d visualization. But not a curved one otherswise your results will look like the image I posted at the beginning of this issue.

[bertjiazheng]

Hi @marcomiglionico94
I think this may be due to the incorrect depth distance we used. You may try two different depths: i) distance from the point to the image plane or ii) distance from point to the camera center.
Best, Jia

Is it possible to know at least what kind of visualization you use for the paper? Did you use Open3D or something else?

We use Open3D to visualize the reconstructed 3D model.

[CYang0515]

Hi, @marcomiglionico94
There is a mistake about the variable name. The returned depth in code is the inverse of depth. So the actually depth is by 1/depth.

[marcomiglionico94]

Hi, @marcomiglionico94 There is a mistake about the variable name. The returned depth in code is the inverse of depth. So the actually depth is by 1/depth.

Hi thanks for your reply. I am not very sure about this. When i return 1/depth the results of the depthmap are very strange since only a small part of the scene is included, the rest is missing. In fact before the values of the depthmap are between 0 and 1, but after i do 1/depth all the values are around 2 or 3, which I don't think is correct. Maybe you mean depth = np.ones_like(depth) - depth ??
Trying to figure out what is the problem

[CYang0515]

We add an example on NYU303 dataset. The actually depth is 1/depth, not depth = np.ones_like(depth) - depth.

[marcomiglionico94]

Using depth = np.ones_like(depth) - depth I was able to obtain the following visualization, which seems correct.

Using 1/depth instead, this is what i was getting for the same image:

[marcomiglionico94]

We add an example on NYU303 dataset. The actually depth is 1/depth, not depth = np.ones_like(depth) - depth.

Hello, I realized that if I use 1/depth it only works if then I divide the values of the depthmap by the max value of it, only to scale the values between 0 and 1 so that I can visualize them. Do you know if that is the correct approach? or is there another way to scale the values of the depthmap between 0 and 1 so that I can visualize it?

[Cli98]

@marcomiglionico94 I wonder if it could be possible for you to briefly discuss how you solve the problem?

[ebrahimkhalilabbasi]

@bertjiazheng and @marcomiglionico94
Could you please share with us the code you implemented to visualise the 3D model?

[HardikJain02]

hey @marcomiglionico94, can you share the code you have implemented for 3D visualization. Also, I am trying to implement this function DisplayMeshLayout in reconstruction.py for my custom images in test.py

It needs these as input def DisplayMeshLayout(ups, downs, attribution, pwalls, pceiling, pfloor, ixy1map, K_inv)

This is I have implemented it in test.py

DisplayMeshLayout(ups, downs, attribution, pwalls=params_layout, pceiling=pceiling, pfloor=pfloor, ixy1map=inputs['ixy1map'][i].cpu().numpy(), K_inv = np.linalg.inv(inputs['intri'][i].cpu().numpy()))

I am getting this error:

Traceback (most recent call last):
  File "/workspace/hardik/NonCuboidRoom/test.py", line 508, in <module>
    test_custom(model, criterion, dataloader, device, cfg)
  File "/workspace/hardik/NonCuboidRoom/test.py", line 370, in test_custom
    DisplayMeshLayout(ups, downs, attribution, pwalls=params_layout, pceiling=pceiling, pfloor=pfloor, ixy1map=inputs['ixy1map'][i].cpu().numpy(), K_inv = np.linalg.inv(inputs['intri'][i].cpu().numpy()))
  File "/workspace/hardik/NonCuboidRoom/models/reconstruction.py", line 669, in DisplayMeshLayout
    poly = empty_polygon.intersection(poly)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/layout/lib/python3.12/site-packages/shapely/geometry/base.py", line 599, in intersection
    return shapely.intersection(self, other, grid_size=grid_size)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/layout/lib/python3.12/site-packages/shapely/decorators.py", line 77, in wrapped
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/layout/lib/python3.12/site-packages/shapely/set_operations.py", line 131, in intersection
    return lib.intersection(a, b, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
shapely.errors.GEOSException: TopologyException: side location conflict at 297 72.826979472140764. This can occur if the input geometry is invalid.