DyLiN: Making Light Field Networks Dynamic

This repository is for the new neral light field (NeLF) method introduced in the following CVPR'23 paper:

DyLiN: Making Light Field Networks Dynamic
Heng Yu ¹, Joel Julin ¹, Zoltán Á Milacski ¹, Koichiro Niinuma ², and László A. Jeni ^1
1 Carnegie Mellon University ² Fujitsu Research of America

Project | ArXiv

Setup

The codebase is based on R2L. Please ensure that line #337 in load_blender.py located in the dataset folder is modified to match the pseudo data format.

Environment

We use the same environment as R2L, using Python 3.9.

Data

For synthetic scenes, we use data from D-NeRF.
For real scenes, we use data from HyperNeRF.

Train DyLiN models

There are several steps in DyLiN training that are similar to R2L.

(1) Train a teacher NeRF model.
(2) Use pretrained teacher NeRF model to generate synthetic data.
(3) Train DyLiN network on the synthetic data -- this step can make our DyLiN model perform comparably to the teacher NeRF model.
(4) Finetune the DyLiN model using the real data -- this step will further boost the performance and make our DyLiN model outperform the teacher NeRF model.

The detailed step-by-step training pipeline is as follows.

Step 1.

For synthetic scenes, please refer to D-NeRF.
For real scenes, please refer to HyperNeRF.

Step 2.

Use the pretrained NeRF model to generate synthetic data (saved in .npy format). For details about the synthetic data, please refer to R2L.
For synthetic scenes:

python tools/gen_pseudo_data_dnerf.py

For real scenes:

python tools/gen_pseudo_data_hypernerf.py

Step 3.

Train R2L model on the synthetic data.
For synthetic scenes (xxx is the set name, e.g. lego, standup and jumpingjacks):

CUDA_VISIBLE_DEVICES=0 python main.py --model_name R2L --config configs/xxx_noview.txt --n_sample_per_ray 16 --netwidth 256 --netdepth 88 --datadir_kd data/nerf_synthetic/xxx_pseudo_images10k_dnerf --n_pose_video 20,1,1 --N_iters 800000 --N_rand 20 --data_mode rays --hard_ratio 0.2 --hard_mul 20 --use_residual --trial.ON --trial.body_arch resmlp --num_worker 8 --warmup_lr 0.0001,200 --cache_ignore data,__pycache__,torchsearchsorted,imgs --screen --project R2L__blender_xxx

For real scenes :

CUDA_VISIBLE_DEVICES=0 python main.py --model_name R2L --config configs/lego_noview.txt --n_sample_per_ray 16 --netwidth 256 --netdepth 88 --datadir_kd data/nerf_synthetic/xxx_pseudo_images10k_hypernerf --n_pose_video 20,1,1 --N_iters 800000 --N_rand 20 --data_mode rays --hard_ratio 0.2 --hard_mul 20 --use_residual --trial.ON --trial.body_arch resmlp --num_worker 8 --warmup_lr 0.0001,200 --cache_ignore data,__pycache__,torchsearchsorted,imgs --screen --project R2L__blender_xxx --hyperdata

Step 4.

Convert original real data (images) to our .npy format.
(1) For synthetic scenes (xxx is the set name, e.g. lego, standup and jumpingjacks):

python tools/convert_original_data_dnerf.py --splits train --datadir data/xxx

(2) For real scenes:

python tools/convert_original_data_hypernerf.py

Finetune the R2L model using real data. (for real scenes, add flag "--hyperdata")

CUDA_VISIBLE_DEVICES=0 python main.py --model_name R2L --config configs/lego_noview.txt --n_sample_per_ray 16 --netwidth 256 --netdepth 88 --datadir_kd data/nerf_synthetic/lego_real_train --n_pose_video 20,1,1 --N_iters 810000 --N_rand 20 --data_mode rays --hard_ratio 0.2 --hard_mul 20 --use_residual --cache_ignore data,__pycache__,torchsearchsorted,imgs --screen --trial.ON --trial.body_arch resmlp --num_worker 8 --warmup_lr 0.0001,200 --save_intermediate_models --pretrained_ckpt Experiments/R2L__blender_lego_SERVER*/weights/ckpt.tar --resume --project R2L__blender_lego__ft

Test DyLiN models

(for real scenes, add flag "--hyperdata")

CUDA_VISIBLE_DEVICES=0 python main.py --model_name R2L --config configs/lego_noview.txt --n_sample_per_ray 16 --netwidth 256 --netdepth 88 --use_residual --cache_ignore data --trial.ON --trial.body_arch resmlp --pretrained_ckpt Experiments/R2L_XXX/weights/ckpt.tar --render_only_fix_pose --testskip 1 --screen --project Test_XXX

Reference

If our work or code helps you, please consider citing our paper. Thank you!

@article{yu2023dylin,
  title={DyLiN: Making Light Field Networks Dynamic},
  author={Yu, Heng and Julin, Joel and Milacski, Zoltan A and Niinuma, Koichiro and Jeni, Laszlo A},
  journal={arXiv preprint arXiv:2303.14243},
  year={2023}
}