This repository contains the 360 light field dataset and its description.
The ultimate goal of this project is to construct Free viewpoint interactive image-based VR system effieicntly by only using a single 360 camera (i.e., using images only)
This is an ongoing project, and some parts will be updated later.
[2022-11-19] Release some parts of dataset and descriptions
The 4-DOF (x,y,z-axis rotation + x-axis movements) interactive image-based VR can be easily made by only using a provided data and the corresponding depths (extracted via the codes & pre-trained models of Yun et.al). The interaction between objects in the scene and virtual object (a bascketball) is calculated based on the corresponding depths. Unlike point-cloud based VR, image-based VR is made by renderring images. Therefore, without additional manual post-processing, image-based VR can yield high quality view without any artefacts (e.g., holes) .
Because it is impossible to capture images of all viewpoint, the images of some viewpoint should be synthesized. Based on the lightfield theory, the images of a new viewpoint can be synthesized efficiently if the images were obtained based on the lightfield structure. More details will be updated later.
The dataset is composed of 360 video and 360 light field, where the differences are summarized in the table below. Both are captured by moving the camera dolly ('Edelkrone' dolly plus) with 360 camera on it, as shown the figure below.
| Model | 360 video | 360 lightfield |
|---|---|---|
| Movement direction | x-axis (column) | x and y-axis (column & row) |
| Lightfield structure | No | Yes |
| Resolution | 512x1024 | 1024x2048 or higher |
| Camera module | Insta 360 pro2 | Go pro fusion |
This dataset is acquired based on the lightfield structure. As shown the figure below (left), the images are captured by moving the 360 cameras along the paths of a pre-calculated grid. The figure below (right) visualizes the dataset. More details will be updated later.
| 360 light field | # of Column | # of Row |
|---|---|---|
| BMW | 392 | 342 |
| Hyundai_1 | 30 | TBU |
| Hyundai_2 | 36 | 60 |
| Room_1 | 18 | 14 |
| Room_2 | 8 | 9 |
This dataset is constructed to augment data for self-supervised learning of 360 depths. The dataset are captured inside the building of INHA university. Because ,ultimately, 360 depth network should learn depths from 'wild 360 videos', the images are captured rather roughly by just moving the cameras along the x-axis (i.e., column) unlike lightfield data.
Below table shows the components of data, where each 'Area' does not necesarily means that they are captured in different places. They are divided just for convenience to balance the dataset usage between other datasets when training the network in a self-supervised manner.
| 360 video | # of Column | # of Row |
|---|---|---|
| Area_1 | 8 - 1 (Column_2) | 0 |
| Area_2 | 10 | 0 |
| Area_3 | 8 | 0 |
| Area_4 | 7 | 0 |
The dataset can be downloaded in this link.
Refer to codes of Yun et.al
TBU
If you are interesed in this dataset, you may also interested in the following papers.
1) Ilwi Yun, Hyuk-Jae Lee, Chae Eun Rhee, "Improving 360 monocular depth estimation via non-local dense prediction transformer and Joint supervised and self-supervised learning", AAAI22
2) S.Lee, H.Jung, and C.E.Rhee, "Data Orchestration for Accelerating GPU-based Light Field Rendering Aiming at a Wide Virtual Space", IEEE Trans. Circuits Syst. Video Technol., Oct. 2021.
- BMW driving centor (https://www.bmw-driving-center.co.kr)
- Hyundai motor studio (https://hyundai.com/kr/ko/brand/motorstudio/seoul/overview)
- Soyoo2 (http://soyoo2.com)
This work was supported by Samsung Research Funding Center of Samsung Electronics under Project Number SRFC-IT1702-54.
Our contributions on codes and data are released under the MIT license.