GANmut: Learning Interpretable Conditional Space for Gamut of Emotions
Stefano d'Apolito1, Danda Pani Paudel1, Zhiwu Huang1, Andres Romero1, Luc Van Gool1,2
1Computer Vision Lab, ETH Zurich, Switzerland, 2PSI, KU Leuven, Belgium
Abstract: Humans can communicate emotions through a plethora of facial expressions, each with its own intensity, nuances and ambiguities. The generation of such variety by means of conditional GANs is limited to the expressions encoded in the used label system. These limitations are caused either due to burdensome labelling demand or the confounded label space. On the other hand, learning from inexpensive and intuitive basic categorical emotion labels leads to limited emotion variability. In this paper, we propose a novel GAN-based framework that learns an expressive and interpretable conditional space (usable as a label space) of emotions, instead of conditioning on handcrafted labels. Our framework only uses the categorical labels of basic emotions to learn jointly the conditional space as well as emotion manipulation. Such learning can benefit from the image variability within discrete labels, especially when the intrinsic labels reside beyond the discrete space of the defined. Our experiments demonstrate the effectiveness of the proposed framework, by allowing us to control and generate a gamut of complex and compound emotions while using only the basic categorical emotion labels during training.
pip install -r requirments.txt to install required dependencies. Finally, install also Pytorch (code has been tested with version 1.4) and Torchvision.
To see an example of how to use GANmut to edit face emotion, install jupyter and open notebook/images_generation.ipynb.
GANmut requires a large and varied training dataset, such as AffectNet. To train it with a custom dataset place the images in the folder dataset/imgs, and add the
file dataset/training.csv. This should have as header subDirectory_filePath,expression, and each row should contain the name of the images (e.g., imgs/Einstein.jpg), and its discrete emotion label, (e.g., 0 for Neutral, 1 for Happy and so on). See a mini example in dataset folder.
Some of the configuraions in scripts "train_ImageNet.sh" and "train_cifar.sh" need to be set according to your experiments. Some of the important parameters are:
| Parameter | Description |
|---|---|
| --c_dim | The number of different categorical emotion in the training set |
| --parametrization | Whether to use the linear or gaussian model |
| --resume_iter | From which iteration resume the training (The relative checkpoint must be preset) |
| --model_save_step | Every model_save_step iterations the model is saved |
Other parameters are described in n "main.py"
# Train GANmut with dataset in in the folder dataset
python -m core.main --dataset_root dataset --imgs_folder imgs --image_size 128 --c_dim 7
--sample_dir samples/samples_linear_2d --log_dir GANmut/logs_linear_2d
--model_save_dir GANmut/models_linear_2d --result_dir GANmut/results_linear_2d
# Test StarGAN using the CelebA dataset
python -m core.main --dataset_root dataset --imgs_folder imgs --image_size 128 --parametrization gaussian
--c_dim 7 --lambda_cls=2. --sample_dir samples/samples_gaussian_2d --log_dir GANmut/logs_gaussian_2d
--model_save_dir GANmut/models_gaussian_2d --result_dir GANmut/results_gaussian_2d For any questions, suggestions, or issues with the code, please contact Stefano at [email protected].
If you find this project helpful, please consider citing us as follows:
@inproceedings{sdapolito2021GANmut,
title = {GANmut: Learning Interpretable Conditional Space for Gamut of Emotions},
author = {d'Apolito, Stefano and
Paudel, Danda Pani and
Huang, Zhiwu and
Romero, Andres and
Van Gool, Luc},
year = {2021},
booktitle = {2021 {IEEE} Conference on Computer Vision and Pattern Recognition, {CVPR} 2021}
}The code has been developed through an extended modification of yunjey's StarGAN