Problems when training A-NeRF in zju_mocap dataset

[Yzmblog]

When I trained A-NeRF in zju_mocap not changing dataloader in this repo, it seems that there will be some artifacts on the floor when visualization. Do you meet the problem when you training on zju_mocap? or whether I did something wrong when training?

[LemonATsu]

Hi, thanks for reaching out.

Since ZJU-Mocap has small pose variation, A-NeRF will just learn everything that won't hurt the reconstruction on training images (it's not constrained by surface model).

If you wish to train an A-NeRF with cleaner background, you can try perturb the background, so that it's A-NeRF cannot capture it easily (and therefore, given up on it as we provided background color to it anyway).

Specifically, see this line here:
https://github.com/LemonATsu/A-NeRF/blob/main/core/dataset.py#L270

You can try replacing it with:

bg = np.random.random(bg.shape).astype(np.float32) * (1 - fg)

So that the background is just a bunch of noise, and A-NeRF cannot forcefully remember this without hurting reconstruction.

I never tried this, but this should work in theory.

[Yzmblog]

Hi, Thanks for your reply! And i have try to replacing the background with noise as you said, but the result is still bad, The disp map is as below:

And i try to change the background to pure black and white, it didn't work out too.
Maybe A-NeRF is not suitable for zju_mocap? or there are some other reasons for this?

[LemonATsu]

"Maybe A-NeRF is not suitable for zju_mocap? or there are some other reasons for this?"
A-NeRF is in general not suitable for dataset with small pose diversity, so yes, zju_mocap is not a suitable dataset for A-NeRF. Like we discussed above: A-NeRF is not constrained by the surface. Novel-view synthesis for this kind of dataset should still be fine to some extend (assuming you train it with multi-view).

If you want to try it a little more, set mask_image=True in the config, along with the noise background mentioned previously. Also, we also have this 'mask supervision setting' that we never use in the original paper, see here:
https://github.com/LemonATsu/A-NeRF/blob/main/run_nerf.py#L245

You can set reg_fn=L1 (or other possible choice here: https://github.com/LemonATsu/A-NeRF/blob/main/core/trainer.py#L159),
and tune the mask loss via reg_coef=.... It basically calculates the loss between the mask and the predicted disp map. You can try with reg_coef=0.001 and see how that works, but I never really tried it.

Again, thanks for reaching out! Let me know if you have further questions.

[Yzmblog]

Hi, thanks for your reply again! And I have try these methods, it's better now. Although still have some artifact, but it's much better! The final val disp image is like this:

Thanks for your discussion and quick reply again!

[LemonATsu]

No problem. I am closing this issue now. Feel free to reopen and comment if you want to discuss further.