A pytorch implementation of dreamfields as described in Zero-Shot Text-Guided Object Generation with Dream Fields.
An example of a generated neural field by prompt "cthulhu" viewed in real-time:
cthulhu.mp4
The code framework is based on torch-ngp.
git clone https://github.com/ashawkey/dreamfields-torch.git
cd dreamfields-torch
pip install -r requirements.txt
# (optional) install the tcnn backbone
pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
By default, we use load
to build the extension at runtime.
However, this may be inconvenient sometimes.
Therefore, we also provide the setup.py
to build each extension:
# install all extension modules
bash scripts/install_ext.sh
# if you want to install manually, here is an example:
cd raymarching
python setup.py build_ext --inplace # build ext only, do not install (only can be used in the parent directory)
pip install . # install to python path (you still need the raymarching/ folder, since this only install the built extension.)
- Ubuntu 20 with torch 1.10 & CUDA 11.3 on a TITAN RTX.
- Windows 10 with torch 1.11 & CUDA 11.3 on a RTX 3070.
Currently, --ff
only supports GPUs with CUDA architecture >= 70
.
For GPUs with lower architecture, --tcnn
can still be used, but the speed will be slower compared to more recent GPUs.
First time running will take some time to compile the CUDA extensions.
# text-guided generation
python main_nerf.py --text "cthulhu" --workspace trial --cuda_ray --fp16
# use the GUI
python main_nerf.py --text "cthulhu" --workspace trial --cuda_ray --fp16 --gui
# [experimental] image-guided generation (also use the CLIP loss)
python main_nerf.py --image /path/to/image --workspace trial --cuda_ray --fp16
check the scripts
directory for more provided examples.
- Mip-nerf is not implemented, currently only the original nerf is supported.
- Sampling poses with an elevation range in [-30, 30] degrees, instead of fixed at 30 degree.
- Use the origin loss.
- 5.18: major update.
- 3.16: basic reproduction.
-
The great paper and official JAX implementation of dreamfields:
@article{jain2021dreamfields, author = {Jain, Ajay and Mildenhall, Ben and Barron, Jonathan T. and Abbeel, Pieter and Poole, Ben}, title = {Zero-Shot Text-Guided Object Generation with Dream Fields}, journal = {arXiv}, month = {December}, year = {2021}, }
-
The GUI is developed with DearPyGui.