API
+This section provides in depth documentation of all the implemented functionality.
+⚒️⚒️ This section is still under construction as I am trying to fix some issues with sphinx and GH pages. ⚒️⚒️
+
+
+
+
+
+
+
+ + +The object network computes :math:`\sigma^{\text{obj}}_i` and :math:`\mathbf{c}^{\text{obj}}_i`, while the +medium network computes :math:`\boldsymbol{\sigma}^{\text{bs}}`, :math:`\boldsymbol{\sigma}^{\text{attn}}` and +:math:`\mathbf{c}^{\text{med}}`. + +The proposal network is used to sample point in regions of the scene that contribute most to the final image. This approach +actually uses two proposal networks that are connected sequentially. More details on the concept of proposal samplers and +how they are optimized during training can be found in :cite:`mipnerf360`. + +For positional encoding, I use Hash Grid Encodings as proposed in :cite:`instant-ngp` and for directional +encoding I use Spherical Harmonics Encoding (SHE) introduced in :cite:`refnerf`. + +The MLPs in the object and medium networks are implemented using `tinycuda-nn
+
+
+
+.. raw:: html
+
+ Click to see renderings of IUI3-RedSea
+ +
+
+
+.. raw:: html
+
+ Click to see renderings of Curasao
+ +
+
+
+.. raw:: html
+
+ Click to see renderings of JapaneseGardens-RedSea
+ ++ +The implementation can also be used to render depthmaps and object weight accumulation maps of the scene. The following videos +show the depthmaps on the left and the object weight accumulation maps on the right. From the object accumulation maps, we +can nicely see, that the model is able to seperate between objects (red hue, indicating high accumulation) and the water (blue +hue, indicating low accumulation). + +.. raw:: html + +
+
+
+.. raw:: html
+
+ Click to see renderings of IUI3-RedSea
+ +
+
+
+.. raw:: html
+
+ Click to see renderings of Curasao
+ +
+
+
+.. raw:: html
+
+ Click to see renderings of JapaneseGardens-RedSea
+ ++ +The model also allows to render only the backscatter of the scene and the clear scene but with attenuation effects. diff --git a/_sources/usage.rst.txt b/_sources/usage.rst.txt new file mode 100644 index 0000000..3fcf162 --- /dev/null +++ b/_sources/usage.rst.txt @@ -0,0 +1,117 @@ +.. _usage-label: + +Usage +===== + +As this approach built-upon nerfstudio, you can use the full functionality of the library. Therefore +please get familiar with the basic commands of this library. You can find the documentation +`here
+ +Note that the image above is cut off. If you run this command on your machine, you can see all the parameters you can specify +when training the implemented Subsea-NeRF. The default options should do fine on most scenes. One thing I strogly recommend is to use the +``--vis wandb`` option, as this will allow you to log training on `W&B
+ +All the panels can be used to inspect the training process. They are very informative and can give you a good sense of the +progress of model training. Make sure to check them out! ✅ + +When specifying ``--vis viewer+wandb``, you can additionally see a live view of the scene during the training process in the +interactive viewer built into nerfstudio. See the documentation `here
+ +.. raw:: html + +
+ +Rendering +********* +After having trained the Subsea-NeRF, you can use it to render videos from arbitrary camera trajectories of the scene. +Make sure to first locate the config.yml of the trained model as you need to pass the path to the rendering script. +This file can be found in the output folder created when training the NeRF. Due to the underlying image formation model +that allows us to seperate between the objects and the water within a scene, you need to +choose the kind of video you want to render. The following options exist: + +- **rgb**: To render the reconstructed scene. +- **J**: To render the clear scene (water effect removed). +- **direct**: To render the attenuated clear scene. +- **bs**: To render the backscatter of the water within the scene. +- **depth**: To render the depthmaps of the scene. +- **accumulation**: To render the object weight accumulation maps of the scene. + +For a detailed explanation of the arguments that can be specified when rendering, you can use the help command: + +.. code-block:: bash + + ns-render --help + +If you want to render out an RGB video of a scene where the camera trajectory is interpolated between the evaluation images of +the dataset, a command looks similar to the following: + +.. code-block:: bash + + ns-render interpolate --load-config
' + + '' + + _("Hide Search Matches") + + "
" + ) + ); + }, + + /** + * helper function to hide the search marks again + */ + hideSearchWords: () => { + document + .querySelectorAll("#searchbox .highlight-link") + .forEach((el) => el.remove()); + document + .querySelectorAll("span.highlighted") + .forEach((el) => el.classList.remove("highlighted")); + localStorage.removeItem("sphinx_highlight_terms") + }, + + initEscapeListener: () => { + // only install a listener if it is really needed + if (!DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS) return; + + document.addEventListener("keydown", (event) => { + // bail for input elements + if (BLACKLISTED_KEY_CONTROL_ELEMENTS.has(document.activeElement.tagName)) return; + // bail with special keys + if (event.shiftKey || event.altKey || event.ctrlKey || event.metaKey) return; + if (DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS && (event.key === "Escape")) { + SphinxHighlight.hideSearchWords(); + event.preventDefault(); + } + }); + }, +}; + +_ready(() => { + /* Do not call highlightSearchWords() when we are on the search page. + * It will highlight words from the *previous* search query. + */ + if (typeof Search === "undefined") SphinxHighlight.highlightSearchWords(); + SphinxHighlight.initEscapeListener(); +}); diff --git a/api.html b/api.html new file mode 100644 index 0000000..3323561 --- /dev/null +++ b/api.html @@ -0,0 +1,115 @@ + + + + + + +
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/genindex.html b/genindex.html
new file mode 100644
index 0000000..f471236
--- /dev/null
+++ b/genindex.html
@@ -0,0 +1,111 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/index.html b/index.html
new file mode 100644
index 0000000..6a4ca9e
--- /dev/null
+++ b/index.html
@@ -0,0 +1,150 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Index
+ +
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/installation.html b/installation.html
new file mode 100644
index 0000000..3cf7168
--- /dev/null
+++ b/installation.html
@@ -0,0 +1,185 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Welcome to the Subsea-NeRF documentation!
+This implementation is an extension of the approach presented in SeaThru-NeRF. +This documentation describes the ideas and underlying equations and network architectures of the approach in detail in the +Introduction. Furthermore, it describes how to install this repository in the section on Installation. +Additionally, it provides some examples on how to use the code in the Usage section and presents some results that can be achieved with it in +the section Results. Finally, it provides an in depth API documentation of the code in the section API.
+
+
+Contents:
+-
+
- Introduction
-
+
- Approach +
+ - Installation
-
+
- Requirements +
+ - Usage + +
- Results
-
+
- Renderings +
+ - API +
Indices and tables
+-
+
- +
- +
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/intro.html b/intro.html
new file mode 100644
index 0000000..a2e221c
--- /dev/null
+++ b/intro.html
@@ -0,0 +1,232 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Installation
+This repository is built upon the nerfstudio +library. This library has a few requirements, with the most important being having access +to a CUDA compatible GPU. Furthermore, for full functionality, it needs colmap, +ffmpeg and tinycuda-nn. +Those requirements might be a bit tricky to install and the installation is system dependent. +The easiest way to install them is to follow the instructions here +up to the point of actually installing nerfstudio.
+For this approach, I extended the nerfstudio library. Therefore please install my fork of +nerfstudio available here. This can be +done via:
+git clone https://github.com/acse-pms122/nerfstudio_dev.git
+cd nerfstudio_dev
+pip install -e .
+Then, clone this repository and install it via:
+cd ..
+git clone https://github.com/ese-msc-2022/irp-pms122.git
+cd irp-pms122
+pip install -e .
+Then, install the command line completion via:
+ns-install-cli
+To check the installation, type:
+ns-train seathru-nerf --help
+If you see the help message, you are good to go! 🚀🚀🚀
+Requirements
+This implementation requires a GPU with a CUDA copatible driver. There are two model configurations as summarised +in the following table:
+Method |
+Description |
+Memory |
+Quality |
+
|---|---|---|---|
|
+Larger model, used to produced results in report |
+~23 GB |
+Best |
+
|
+Smaller model |
+~7 GB |
+Good |
+
I recommend to use the seathru-nerf method as it was used to experiment and produce the results presented in the paper.
+The seathru-nerf-lite still produces good results, but has not been tested on all scenes. If you happen to run into a
+CUDA_OUT_MEMORY_ERROR it is a sign that the available VRAM on the GPU is not enough. You can either use the smaller
+model, decrease the batch size, do both or upgrade to a better GPU.
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/objects.inv b/objects.inv
new file mode 100644
index 0000000..e149e37
Binary files /dev/null and b/objects.inv differ
diff --git a/results.html b/results.html
new file mode 100644
index 0000000..058c0b6
--- /dev/null
+++ b/results.html
@@ -0,0 +1,147 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Introduction
+With Neural Radiance Fields (NeRFs), we can store a 3D scene as a continuous function. +This idea was first introduced in the original NeRF publication [5]. +Since then, the field experienced many advancements. Some of them even in the subsea domain. +However, these advancements still have some limitations. This implementation adresses some of those limitations +and provides a modular and documented implementation of a subsea specific NeRF that allows for easy modifications and experiments.
+Approach
+The fundamental principle underlying NeRFs is to represent a scene as a continuous function that maps a position, +\(\mathbf{x} \in \mathbb{R}^{3}\), and a viewing direction, \(\boldsymbol{\theta} \in \mathbb{R}^{2}\), +to a color \(\mathbf{c} \in \mathbb{R}^{3}\) and volume density \(\sigma\). We can approximate this +continuous scene representation with a simple Multi Layer Perceptron (MLP). +\(F_{\mathrm{\Theta}} : (\mathbf{x}, \boldsymbol{\theta}) \to (\mathbf{c},\sigma)\).
+It is common to also use positional and directional encodings to improve the performance of NeRF approaches. Furthermore, +there are various approaches in order to sample points in regions of a scene that are relevant to the final image. A detailed +explanation of the exact implemented architecture is given in the Network architecture section.
+Image formation model
+The authors of [3] combine the fundamentals of NeRFs with the following underwater image formation model +proposed in [1]:
+
+\[I = \overbrace{\underbrace{J}_{\text{colour}} \cdot \underbrace{(e^{-\beta^D(\mathbf{v}_D)\cdot z})}_{\text{attenuation}}}^{\text{direct}} + \overbrace{\underbrace{B^\infty}_{\text{colour}} \cdot \underbrace{(1 - e^{-\beta^B(\mathbf{v}_B)\cdot z})}_{\text{attenuation}}}^{\text{backscatter}}\]
+\(I\) …………… Image
+\(J\) …………… Clear image (without any water effects like attenuation or backscatter)
+\(B^\infty\) ……….. Backscatter water colour at depth infinity
+\(\beta^D(\mathbf{v}_D)\) … Attenuation coefficient [1]
+\(\beta^B(\mathbf{v}_B)\) … Backscatter coefficient [1]
+\(z\) …………… Camera range
+This image formation model allows the model to seperate between the clean scene and the water effects. This is very useful +since it allows for filtering out of water effects from a scene. Some results where this was achieved are shown in the +Results section.
+Rendering equations
+As NeRFs require a discrete and differentiable volumetric rendering equation, the authors of [3] propose +the following formulation:
+
+\[\hat{\boldsymbol{C}}(\mathbf{r}) = \sum_{i=1}^N \hat{\boldsymbol{C}}^{\text{obj}}_i(\mathbf{r}) + \sum_{i=1}^N \hat{\boldsymbol{C}}^{\text{med}}_i(\mathbf{r})\]
+This equation features an object and a medium part contributing towards the final rendered pixel +colour \(\hat{\boldsymbol{C}}(\mathbf{r})\). Those two components are given by:
+
+\[\hat{\boldsymbol{C}}^{\text{obj}}_i(\mathbf{r}) = T^{\text{obj}}_i \cdot \exp (-\boldsymbol{\sigma}^{\text{attn}} t_i) \cdot \left(1 - \exp({-\sigma^{\text{obj}}_i \delta_i})\right) \cdot \mathbf{c}^{\text{obj}}_i\]
+
+\[\hat{\boldsymbol{C}}^{\text{med}}_i(\mathbf{r}) = T^{\text{obj}}_i \cdot \exp (-\boldsymbol{\sigma}^{\text{bs}} t_i) \cdot \left(1 - \exp({-\boldsymbol{\sigma}^{\text{bs}} \delta_i})\right) \cdot \mathbf{c}^{\text{med}}\]
+, with
+
+\[T^{\text{obj}}_i = \exp\left(-\sum_{j=0}^{i-1}\sigma^{\text{obj}}_j\delta_j\right)\]
+The above equations contain five parameters that are used to describe the underlying scene: +object density \(\sigma^{\text{obj}}_i \in \mathbb{R}^{1}\), object colour +\(\mathbf{c}^{\text{obj}}_i \in \mathbb{R}^{3}\), backscatter density +\(\boldsymbol{\sigma}^{\text{bs}} \in \mathbb{R}^{3}\), attenuation density +\(\boldsymbol{\sigma}^{\text{attn}} \in \mathbb{R}^{3}\), and medium colour +\(\mathbf{c}^{\text{med}} \in \mathbb{R}^{3}\).
+I use the network discussed below to compute those five parameters that parametrize the underlying scene.
+Network architecture
+The network implemented for this approach has the following architecture:
+
+The object network computes \(\sigma^{\text{obj}}_i\) and \(\mathbf{c}^{\text{obj}}_i\), while the +medium network computes \(\boldsymbol{\sigma}^{\text{bs}}\), \(\boldsymbol{\sigma}^{\text{attn}}\) and +\(\mathbf{c}^{\text{med}}\).
+The proposal network is used to sample point in regions of the scene that contribute most to the final image. This approach +actually uses two proposal networks that are connected sequentially. More details on the concept of proposal samplers and +how they are optimized during training can be found in [2].
+For positional encoding, I use Hash Grid Encodings as proposed in [4] and for directional +encoding I use Spherical Harmonics Encoding (SHE) introduced in [6].
+The MLPs in the object and medium networks are implemented using tinycuda-nn for +performance reasons.
+Footnotes
+ +References
+
+
+
+
+
+[1]
+
+Derya Akkaynak and Tali Treibitz. A revised underwater image formation model. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 6 2018.
+
+[2]
+
+Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: unbounded anti-aliased neural radiance fields. CVPR, 2022.
+
+[3]
+(1,2)
+
+Deborah Levy, Amit Peleg, Naama Pearl, Dan Rosenbaum, Derya Akkaynak, Simon Korman, and Tali Treibitz. Seathru-nerf: neural radiance fields in scattering media. 2023. arXiv:2304.07743.
+
+[4]
+
+Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., 41(4):102:1–102:15, July 2022. doi:10.1145/3528223.3530127.
+
+[5]
+
+Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: representing scenes as neural radiance fields for view synthesis. In ECCV. 2020.
+
+[6]
+
+Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T. Barron, and Pratul P. Srinivasan. Ref-NeRF: structured view-dependent appearance for neural radiance fields. CVPR, 2022.
+
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/search.html b/search.html
new file mode 100644
index 0000000..7cf0f58
--- /dev/null
+++ b/search.html
@@ -0,0 +1,126 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Results
+This section shows off some results that can be achieved with this repository. To get a sense of the capabilities and +limitations of the implemented approach, have a look at some of them!🔍
+Renderings
+This are some results that show the reconstructed scene on the left and the clean scene on the right.
+
+
Click to see renderings of IUI3-RedSea
+ +
+
Click to see renderings of Curasao
+ +
+
Click to see renderings of JapaneseGardens-RedSea
+ +The implementation can also be used to render depthmaps and object weight accumulation maps of the scene. The following videos +show the depthmaps on the left and the object weight accumulation maps on the right. From the object accumulation maps, we +can nicely see, that the model is able to seperate between objects (red hue, indicating high accumulation) and the water (blue +hue, indicating low accumulation).
+
+
Click to see renderings of IUI3-RedSea
+ +
+
Click to see renderings of Curasao
+ +
+
Click to see renderings of JapaneseGardens-RedSea
+ +The model also allows to render only the backscatter of the scene and the clear scene but with attenuation effects.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/searchindex.js b/searchindex.js
new file mode 100644
index 0000000..1ba95e5
--- /dev/null
+++ b/searchindex.js
@@ -0,0 +1 @@
+Search.setIndex({"alltitles": {"API": [[0, null]], "Approach": [[3, "approach"]], "Contents:": [[1, null]], "Datasets": [[5, "datasets"]], "Example dataset": [[5, "example-dataset"]], "Image formation model": [[3, "image-formation-model"]], "Indices and tables": [[1, "indices-and-tables"]], "Installation": [[2, null]], "Introduction": [[3, null]], "Network architecture": [[3, "network-architecture"]], "Rendering": [[5, "rendering"]], "Rendering equations": [[3, "rendering-equations"]], "Renderings": [[4, "renderings"]], "Requirements": [[2, "requirements"]], "Results": [[4, null]], "Training": [[5, "training"]], "Usage": [[5, null]], "Using your own datasets": [[5, "using-your-own-datasets"]], "Welcome to the Subsea-NeRF documentation!": [[1, null]]}, "docnames": ["api", "index", "installation", "intro", "results", "usage"], "envversion": {"sphinx": 62, "sphinx.domains.c": 3, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 9, "sphinx.domains.index": 1, "sphinx.domains.javascript": 3, "sphinx.domains.math": 2, "sphinx.domains.python": 4, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinxcontrib.bibtex": 9}, "filenames": ["api.rst", "index.rst", "installation.rst", "intro.rst", "results.rst", "usage.rst"], "indexentries": {}, "objects": {}, "objnames": {}, "objtypes": {}, "terms": {"": [3, 5], "0": 3, "07743": 3, "1": 3, "10": 3, "102": 3, "1145": 3, "15": 3, "2": 3, "2018": 3, "2020": 3, "2022": [2, 3], "2023": 3, "23": 2, "2304": 3, "3": 3, "3528223": 3, "3530127": 3, "360": 3, "3d": 3, "4": 3, "41": 3, "5": 3, "6": 3, "7": 2, "A": 3, "As": [3, 5], "For": [2, 3, 5], "If": [2, 5], "In": 3, "It": [3, 5], "On": 5, "One": 5, "The": [2, 3, 4, 5], "Then": 2, "There": 2, "To": [2, 4, 5], "With": 3, "_": 3, "_b": 3, "_d": 3, "_i": 3, "_j": 3, "abl": 4, "abov": [3, 5], "access": 2, "accumul": [4, 5], "achiev": [1, 3, 4], "acm": 3, "acs": 2, "actual": [2, 3], "ad": 5, "addition": [1, 5], "adress": 3, "advanc": 3, "after": 5, "akkaynak": 3, "alex": 3, "alexand": 3, "alias": 3, "all": [0, 2, 3, 5], "allow": [3, 4, 5], "also": [3, 4, 5], "am": 0, "ambient": 3, "amit": 3, "an": [1, 3, 5], "ani": 3, "anti": 3, "api": [1, 5], "appear": 3, "approach": [1, 2, 4, 5], "approxim": 3, "ar": [2, 3, 4, 5], "arbitrari": 5, "architectur": 1, "argument": 5, "arxiv": 3, "attenu": [3, 4, 5], "attn": 3, "author": 3, "avail": 2, "b": [3, 5], "backscatt": [3, 4, 5], "barron": 3, "basic": 5, "batch": 2, "beam": 3, "been": 2, "being": 2, "below": [3, 5], "ben": 3, "best": 2, "beta": 3, "better": 2, "between": [3, 4, 5], "bit": 2, "blue": 4, "boldsymbol": 3, "both": 2, "built": [2, 5], "c": 3, "camera": [3, 5], "can": [1, 2, 3, 4, 5], "capabl": 4, "cd": 2, "cdot": 3, "check": [2, 5], "choos": 5, "christoph": 3, "clean": [3, 4], "clear": [3, 4, 5], "cli": 2, "click": 4, "clone": 2, "code": 1, "coeffici": 3, "colmap": 2, "color": 3, "colour": 3, "com": 2, "combin": 3, "command": [2, 5], "common": 3, "compat": 2, "complet": 2, "compon": 3, "comput": 3, "concept": 3, "confer": 3, "config": 5, "configur": 2, "connect": 3, "construct": 0, "contain": 3, "continu": 3, "contribut": 3, "copat": 2, "creat": 5, "cuda": 2, "cuda_out_memory_error": 2, "curasao": 4, "cut": 5, "cvpr": 3, "d": 3, "dan": 3, "data": 5, "dataset": 1, "deborah": 3, "decreas": 2, "default": 5, "delta_i": 3, "delta_j": 3, "densiti": 3, "depend": [2, 3], "depth": [0, 1, 3, 5], "depthmap": [4, 5], "derya": 3, "describ": [1, 3], "descript": 2, "desired_path_for_output": 5, "detail": [1, 3, 5], "differenti": 3, "direct": [3, 5], "discret": 3, "discuss": 3, "do": [2, 5], "document": [0, 3, 5], "doi": 3, "domain": 3, "done": 2, "dor": 3, "download": 5, "driver": 2, "due": 5, "dure": [3, 5], "e": [2, 3], "easi": 3, "easiest": 2, "eccv": 3, "effect": [3, 4, 5], "either": 2, "encod": 3, "enough": 2, "equat": 1, "es": 2, "evalu": 5, "evan": 3, "even": 3, "exact": 3, "exampl": 1, "exist": 5, "exp": 3, "experi": [2, 3, 5], "experienc": 3, "explan": [3, 5], "extend": 2, "extens": 1, "f_": 3, "familiar": 5, "featur": [3, 5], "few": 2, "ffmpeg": 2, "field": 3, "file": 5, "filter": 3, "final": [1, 3], "find": 5, "fine": 5, "first": [3, 5], "five": 3, "fix": 0, "folder": 5, "follow": [2, 3, 4, 5], "footnot": 3, "forget": 5, "fork": 2, "format": 5, "formul": 3, "found": [3, 5], "from": [3, 4, 5], "full": [2, 5], "function": [0, 2, 3, 5], "fundament": 3, "furthermor": [1, 2, 3], "gb": 2, "get": [4, 5], "gh": 0, "git": 2, "github": 2, "give": 5, "given": 3, "go": 2, "good": [2, 5], "gpu": 2, "graph": 3, "graphic": 3, "grid": 3, "guid": 5, "ha": [2, 3], "happen": 2, "harmon": 3, "hash": 3, "hat": 3, "have": [2, 3, 4, 5], "hedman": 3, "help": [2, 5], "here": [2, 5], "high": 4, "how": [1, 3, 5], "howev": 3, "http": 2, "hue": 4, "i": [0, 1, 2, 3, 4, 5], "idea": [1, 3], "ieee": 3, "imag": 5, "implement": [0, 1, 2, 3, 4, 5], "import": 2, "improv": 3, "includ": 5, "index": 1, "indic": 4, "infin": 3, "inform": 5, "infti": 3, "inspect": 5, "instal": 1, "instant": 3, "instruct": [2, 5], "interact": 5, "interpol": 5, "introduc": 3, "introduct": 1, "irp": 2, "issu": 0, "iui3": [4, 5], "j": [3, 5], "japanesegarden": 4, "jonathan": 3, "juli": 3, "kei": 5, "keller": 3, "kind": 5, "korman": 3, "larger": 2, "layer": 3, "left": [3, 4], "levi": 3, "librari": [2, 5], "light": 3, "like": [3, 5], "limit": [3, 4], "line": 2, "lite": 2, "live": 5, "load": 5, "locat": 5, "log": 5, "look": [4, 5], "low": 4, "machin": 5, "make": 5, "mani": 3, "map": [3, 4, 5], "mathbb": 3, "mathbf": 3, "mathrm": 3, "matthew": 3, "med": 3, "media": 3, "medium": 3, "memori": 2, "messag": 2, "method": 2, "might": 2, "mildenhal": 3, "mip": 3, "mlp": 3, "model": [2, 4, 5], "modif": 3, "modular": 3, "more": 3, "most": [2, 3, 5], "msc": 2, "multi": 3, "multiresolut": 3, "my": 2, "m\u00fcller": 3, "n": [2, 3, 5], "naama": 3, "name": 5, "need": [2, 5], "nerf": [2, 3, 5], "nerfstudio": [2, 5], "nerfstudio_dev": 2, "network": 1, "neural": 3, "ng": 3, "nice": 4, "nn": [2, 3], "note": 5, "obj": 3, "object": [3, 4, 5], "off": [4, 5], "onli": 4, "optim": 3, "option": 5, "order": 3, "origin": 3, "out": [3, 5], "output": 5, "overbrac": 3, "overview": 5, "p": 3, "page": [0, 1, 5], "panel": 5, "paper": 2, "paramet": [3, 5], "parametr": 3, "part": 3, "pass": 5, "path": 5, "path_to_config": 5, "path_to_seathru_nerf_dataset": 5, "pattern": 3, "pearl": 3, "peleg": 3, "perceptron": 3, "perform": 3, "peter": 3, "physic": 3, "pip": 2, "pixel": 3, "pleas": [2, 5], "pms122": 2, "point": [2, 3, 5], "posit": 3, "pratul": 3, "present": [1, 2], "primit": 3, "principl": 3, "proceed": 3, "process": 5, "produc": 2, "progress": 5, "propos": 3, "provid": [0, 1, 3, 5], "public": 3, "qualiti": 2, "r": 3, "radianc": 3, "ramamoorthi": 3, "rang": 3, "ravi": 3, "reason": 3, "recognit": 3, "recommend": [2, 5], "reconstruct": [4, 5], "red": 4, "redsea": [4, 5], "ref": 3, "refer": [3, 5], "reflect": 3, "region": 3, "relev": 3, "remov": 5, "ren": 3, "render": 1, "report": 2, "repositori": [1, 2, 4], "repres": 3, "represent": 3, "requir": [1, 3], "respons": 3, "result": [1, 2, 3, 5], "revis": 3, "rgb": 5, "right": [3, 4], "rosenbaum": 3, "run": [2, 5], "sampl": 3, "sampler": 3, "scatter": 3, "scene": [2, 3, 4, 5], "schi": 3, "script": 5, "search": 1, "seathru": [1, 2, 3, 5], "section": [0, 1, 3, 4, 5], "see": [2, 4, 5], "sens": [4, 5], "seper": [3, 4, 5], "sequenti": 3, "setahru": 5, "she": 3, "should": 5, "show": 4, "shown": 3, "sigma": 3, "sign": 2, "similar": 5, "simon": 3, "simpl": 3, "sinc": 3, "size": 2, "smaller": 2, "some": [0, 1, 3, 4, 5], "someth": 5, "specif": 3, "specifi": 5, "spectral": 3, "spectrum": 3, "spheric": 3, "sphinx": 0, "srinivasan": 3, "start": 5, "still": [0, 2, 3], "store": 3, "strogli": 5, "structur": 3, "subsea": [3, 5], "sum_": 3, "summaris": 2, "sure": 5, "synthesi": 3, "synthet": 5, "system": 2, "t": 3, "t_i": 3, "tabl": 2, "tali": 3, "tancik": 3, "team": 5, "tensorboard": 5, "test": 2, "text": 3, "thei": [3, 5], "them": [2, 3, 4, 5], "therefor": [2, 5], "theta": 3, "thi": [0, 1, 2, 3, 4, 5], "thing": 5, "thoma": 3, "those": [2, 3], "through": 5, "time": 5, "tinycuda": [2, 3], "todd": 3, "toward": 3, "train": [1, 2, 3], "trajectori": 5, "tran": 3, "treibitz": 3, "tricki": 2, "try": 0, "two": [2, 3], "type": 2, "u": 5, "unbound": 3, "under": 0, "underbrac": 3, "underli": [1, 3, 5], "underwat": 3, "up": 2, "upgrad": 2, "upon": [2, 5], "us": [1, 2, 3, 4], "usag": 1, "v": 3, "variou": 3, "verbin": 3, "veri": [3, 5], "vi": 5, "via": 2, "video": [4, 5], "view": [3, 5], "viewer": 5, "vision": 3, "volum": 3, "volumetr": 3, "vram": 2, "w": 5, "wa": [2, 3, 5], "wai": 2, "wandb": 5, "want": 5, "water": [3, 4, 5], "wavelength": 3, "we": [3, 4], "weight": [4, 5], "well": 5, "when": 5, "where": [3, 5], "which": 3, "while": 3, "within": 5, "without": 3, "x": 3, "yml": 5, "you": [2, 5], "z": 3, "zickler": 3}, "titles": ["API", "Welcome to the Subsea-NeRF documentation!", "Installation", "Introduction", "Results", "Usage"], "titleterms": {"api": 0, "approach": 3, "architectur": 3, "content": 1, "dataset": 5, "document": 1, "equat": 3, "exampl": 5, "format": 3, "imag": 3, "indic": 1, "instal": 2, "introduct": 3, "model": 3, "nerf": 1, "network": 3, "own": 5, "render": [3, 4, 5], "requir": 2, "result": 4, "subsea": 1, "tabl": 1, "train": 5, "us": 5, "usag": 5, "welcom": 1, "your": 5}})
\ No newline at end of file
diff --git a/usage.html b/usage.html
new file mode 100644
index 0000000..d8ff0cb
--- /dev/null
+++ b/usage.html
@@ -0,0 +1,194 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Usage
+As this approach built-upon nerfstudio, you can use the full functionality of the library. Therefore +please get familiar with the basic commands of this library. You can find the documentation +here.
+Below, you can find some examples of how to use this library. I will run you through basic training and rendering commands. +Additionally I will include an example of how to use the feature of adding synthetic water to a scene.
+Datasets
+Example dataset
+The Seathru-NeRF dataset was used for all examples and results in this documentation. It is a good starting point to experiment +with and it can be downloaded here.
+Using your own datasets
+If you want to use your own dataset, please +refer to the guide here.
+Training
+🌟🌟🌟It’s time to train your first subsea-NeRF🌟🌟🌟
+To get an overview of the training options, you can use the help command:
+ns-train setahru-nerf --help
+The output should look something like this:
+
+Note that the image above is cut off. If you run this command on your machine, you can see all the parameters you can specify
+when training the implemented Subsea-NeRF. The default options should do fine on most scenes. One thing I strogly recommend is to use the
+--vis wandb option, as this will allow you to log training on W&B. (there is also an option for
+tensorboard) If you specify this option, do not forget to provide your API key as well.
If you want to train the implemented subsea-NeRF model on the IUI3-RedSea scene of the Seathru-NeRF dataset, that +can be downloaded following the instructions above, you can use the following command:
+ns-train seathru-nerf --vis wandb --data <path_to_Seathru_NeRF_dataset>/IUI3-RedSea
+On your wandb page you can then see. something that looks like the following:
+
+All the panels can be used to inspect the training process. They are very informative and can give you a good sense of the +progress of model training. Make sure to check them out! ✅
+When specifying --vis viewer+wandb, you can additionally see a live view of the scene during the training process in the
+interactive viewer built into nerfstudio. See the documentation here
+and this instructional video to use the viewer (provided by the nerfstudio team):
Rendering
+After having trained the Subsea-NeRF, you can use it to render videos from arbitrary camera trajectories of the scene. +Make sure to first locate the config.yml of the trained model as you need to pass the path to the rendering script. +This file can be found in the output folder created when training the NeRF. Due to the underlying image formation model +that allows us to seperate between the objects and the water within a scene, you need to +choose the kind of video you want to render. The following options exist:
+-
+
rgb: To render the reconstructed scene.
+J: To render the clear scene (water effect removed).
+direct: To render the attenuated clear scene.
+bs: To render the backscatter of the water within the scene.
+depth: To render the depthmaps of the scene.
+accumulation: To render the object weight accumulation maps of the scene.
+
For a detailed explanation of the arguments that can be specified when rendering, you can use the help command:
+ns-render --help
+If you want to render out an RGB video of a scene where the camera trajectory is interpolated between the evaluation images of +the dataset, a command looks similar to the following:
+ns-render interpolate --load-config <path_to_config.yml> --rendered-output-names rgb --output-path <desired_path_for_output>
+Some results of example renderings are provided in the Results section.
+