We propose multiple methods for improving state-of-the-art GAN-based video synthesis approaches. We show that GANs using 3D-convolutions for video generation can easily be extended to predicting coherent depth maps alongside RGB frames, but results indicate that this does not improve RGB accuracy if depth is available. We further propose critic-correction, a method for improving videos generated by latent space curve fitting. Additionally, we study the effect of Principal Component Analysis as well as different backprojection methods on the quality of generated videos.
A full report on our work can be found at ./Report/ViDGAN.pdf
We extended an existing 3D-convolution GAN architecture to study the effect of depth information on video synthesis.
Our code is based on the FutureGAN implementation by Aigner and Körner and can be found at ./FutureGAN_Depth. For training, we used Google Colaboratory, the training script is located at FutureGAN_Depth/Training_Depth.ipynb.
We extended the FutureGAN architecture by incorporating normalized dense depth maps as an additional input and/or output channel.
We conducted an ablation study to study the effect of depth information on generated videos. A sample sequence from the test set can be found below:
| Ground Truth | RGB → RGB | RGB-D → RGB | RGB → RGB-D | RGB-D → RGB-D |
|---|---|---|---|---|
 |
 |
 |
 |
 |
 |
 |
 |
The following scores highlight our findings that depth information does not improve the quality of the generated RGB videos. Depth predictions, however, can be learned with almost no additonal cost and are fairly accuracte, even when no depth information is available at test time.
In our second approach, we propose a video generation pipeline leveraging smooth latent spaces.
Our code can be found at ./LSVG. The generator network wgan_ref.py is an extension of DCGAN, the Google Colab scripts used for training the elements of our pipeline can be found at:
- LSVG/Train_WGAN_GP.ipynb
- LSVG/Pipeline.ipynb
- LSVG/Train_Pnet.ipynb
We propose the following steps for synthesizing videos conditioned on a set of input frames:
- Backprojecting a set of input frames onto the latent manifold of a previously trained GAN
- Projection net with structure similar to WGAN critic
- Numerical optimizer initializes with Projection-net output
- Curve-fitting to the projected samples in latent space via function regression
- Sampling of new points on the fitted curves
- Generation of images by feeding the sampled points through the generator of the GAN
- Critic-correction to post-process the generated frames and obtain sharp frames
While our approach worked well in general, the latent space of our trained GAN was not perfectly smooth, so some digit changes are observed in the generated sequences. Below are four sequences (10 generated frames) with 3 input frames from the testset each.
