Project EmoTorch is built for the Facebook Hackathon Contest in the Artificial Intelligence Track using the mentioned tool PyTorch.
Contributors
- Ahmed Hamido - Egypt
- Nathan Curtis - USA
- Yashika Sharma - India
Face Emotion Recognition project aims at predicting the emotions of the user based on their facial image.
The motivation for this project came from the recommendation systems. Often we see products recommended based on the user's search history, watch history , purchase history etc. These sort of systems depend on the past history and records. We with this project want to use real-time recommendation system. The system will use our model to predict the emotion of the user based on his facial expressions and will recommend products to suit the mood of the user.
- Dataset
- Used Libraries
- Network Architecture
- Model
- Hyperparameters
- Performance
- Further Additions
- What is Next
- References
- License
The model is built entirely on the publicly available datasets. Since, the field Facial Emotion Recognition(FER) is not much developed yet we had very limited available datasets. Based on our research, we chose the FER dataset.
The data consists of 48x48 pixel grayscale images of faces. The faces have been automatically registered so that the face is more or less centered and occupies about the same amount of space in each image. The task is to categorize each face based on the emotion shown in the facial expression in to one of seven categories.
| Class | Label |
|---|---|
| 0 | angry |
| 1 | disgust |
| 2 | fear |
| 3 | happy |
| 4 | neutral |
| 5 | sad |
| 6 | surprise |
The dataset is downloaded from Kaggle.
The reason behind choosing this dataset among others-
- It has images categorized in one of the seven emotions.
- Is publicly available
- The length of the dataset is suitble for our task with
- Training set: 28,709 examples.
- Test set: 3,589 examples. -Validation set: 3,589 examples.
The file train.csv contains two columns, "emotion" and "pixels". The "emotion" column contains a numeric code ranging from 0 to 6, inclusive, for the emotion that is present in the image. The "pixels" column contains a string surrounded in quotes for each image. The contents of this string a space-separated pixel values in row major order. test.csv contains only the "pixels" column and our task is to predict the emotion column.
This dataset was prepared by Pierre-Luc Carrier and Aaron Courville, as part of an ongoing research project.
-
PyTorch- This is the main library used in this project. The model is entirely build on PyTorch including all the image transformation, transfer learning, training, testing.
-
NumPy- Used for pixels manipulation
-
Matplotlib- To plot images and loss plots.
-
Tensorboard- Used for creating the dashboard for accuracy and loss plots
The motivation of using transfer learning for out task came after we implemented a Deep Neural Network from Scratch. The model build from scratch gives accuracy only around 18%-20%. We can boost the accuracy with the help of transfer learning.
PyTorch's models module has variety of pretrained networks which can be easily downloaded.
In our project, we tried multiple networks before settling to VGG19.
Initially we used VGG16 which gave us accuracy below 40% followed by ResNet50 with 41% and DenseNet101 with 42.5%. VGG19 yields an accuracy of 46% which is better than all other pretrained model.
Therefore, we decided to choose VGG19 for the implemetation.
The pretrained models are trained on the ImageNet Dataset classifying 1000 classes. Our task is to classify the face expression in one of the 7 classes, so we modified the classification layer and replaced it with our Network. The final model is a combination of transfer learning and custom trained classifier.
The training was done with images in a batch of 64. The learning rate was set nethier too large nor too small that to 0.0001.
To avoid overfitting we used dropout layers in between hidden layers followed by ReLU activation function. The number of epochs are 25 and the hidden layers are 1024 because the model did not improve beyond that.
The optimzer chosen is Adam.
The model is completely built on the public free available dataset in contrast to the commercial FER projects that use large datasets with high resolution quality. However, the model is capable of being trained on any dataset and predicting the accurate emotions. Currently the accuracy is 46% which can be improved with diverse datasets.
- Training Loss
-Some screenshots from Tensorboard Dashboard:
-Display Results
The distribution of samples per category in the FER dataset is not balanced. The category disguist is least represented with only 547 samples whereas the category happiness is most represented with 8989 samples.
Future Scope lies in augmentation. Multiple balancing techniques can be used to present equal number of apparations per category which will result into higher accuracy.
The model is ready and predicts accurate emotions based on the image of a person's face. Our aim to build this project is to merge it with Facebook's product recommendation system.
The predicted labels will be sent to the recommendation system which will inturn predict the most likely products.
The front camera of a cell phone or laptop will capture the face of the person on consent while browsing Facebook Feed. Alternatively, a selfie can be fed. This image is sent to our model which predicts the emotion of the user. This emotion is sent to the recommendation system which predicts the products based on those emotions. These emotions can also be used to recommend things like Place to Visit, Food to Eat, Song to Listen and Movie to watch.
This will help in better product recommendation which in turn will increase the probability of customer buying or clicking on a product.
Article Reference for EmoTorch can be found here
Thanks to Mahdi Ashraf for reviewing the data setup
Inspired by 2020 Facebook AI Hackathon Contest.