This project is my Capstone project for the ML-bootcamp (with Alexey Grigorev). The goal is to gather some experience in various steps of ML pipeline.
I selected the FER2013 (Facial Expression Recognition) dataset in order to try solving an Image Multiclass Classification problem. Such a problem is interesting because it tries to classify isolated faces among several expressions (7 on this dataset).
Such models can probably be used in:
- the virtual reality or the virtual meetings with avatars (to transcribe the facial expression to the other participants),
- the video games (so that the game can know if it must adjust to the player or not)
- the interactive street advertisements (to evaluate how the public react when exposed to a given product),
- the shows (to evaluate the appreciation of the movie, theater show etc.)
- maybe even to school (so that teachers can know if a kid has a problem, is distracted etc...)
- ...
The full set consists of 35,887 samples, and the data consists of 48x48 pixel grayscale images of faces. The faces have been automatically registered so that the face is more or less centered and occupies about the same amount of space in each image.
It has two columns, "emotion" and "pixels". The "emotion" column contains a numeric code ranging from 0 to 6, inclusive, for the emotion that is present in the image. The "pixels" column contains a string surrounded in quotes for each image. The contents of this string a space-separated pixel values in row major order.
This dataset was prepared by Pierre-Luc Carrier and Aaron Courville, as part of an ongoing research project. The dataset is too large to be hosted on github, but it can be downloaded from Kaggle:
- https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge
- https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data?select=fer2013.tar.gz
One can try the application here: https://ml-fer2013.herokuapp.com/input
>> git clone https://github.com/Valkea/ML_Facial_Expression_Recognition.git
>> cd ML_Facial_Expression_RecognitionInstall pipenv if needed
>> pip install pipenvStart the pipenv virtual environment:
>> pipenv shell(If it doesnt' work due to the Python 3.8 requirement, you can edit the Pipfile with you own version. I think it should work with any Python3 version has I didn't used any specific function or method.)
Install the production + developement dependencies:
(venv) >> pipenv install
(venv) >> pipenv install --devNow let's collect the dataset and the models from the DVC hosting:
>> dvc remote add origin https://dagshub.com/Valkea/ML_Facial_Expression_Recognition.dvc
>> dvc pull -r originIf it doesn't works for you, you can grab the .csv file here:
>> jupyter notebook
or
>> jupyter notebook notebook-01-EDA.ipynb*(The notebook menu was created using some HTML, and the GitHub preview doesn't render all of them. But it should work locally)
This project focused on Convolutional Neural Network (CNN) architectures, but several other classifications methods were also tried for comparisons.
The exported model can be tested using:
- the 'Load Models.ipynb' notebook
- the 'test_camera.py' python script (you need a working webcam, but it works with several users at the same time)
- the 'test_image_local.py' python script (just in case you don't have a webcam)
- the 'test_image_remote.py' python script (it sends a picture to the AWS lambda function in order to get a prediction from the remotly hosted model)
Start Flask development server:
(venv) >> python fer2013_server.pyStop with CTRL+C (once the tests are done from another terminal...)
One can check that the server is running by opening the following url: http://0.0.0.0:5000/
Then by submitting various predefined pictures, various results should be displayed.
For conveniance, I created two scripts in order to test prediction by sending a string (same as in the dataset) or a picture.
>> python test_from_string.py This should return an "Angry" label for the given face.
>> python test_from_image.py medias/Angry.jpg
>> python test_from_image.py medias/Fear.jpg
>> python test_from_image.py medias/Sad.jpg
>> python test_from_image.py medias/Happy.jpg
>> python test_from_image.py medias/Neutral.jpg(venv) >> python test_from_camera.py /!\ You need to run it from the pipenv shell
Press any key to stop the program.
This script is built using the MTCNN library to locate the face in each picture and then the captured face is used to predict expression from my FER model.
It can handle several people at the same time (up to 3), but I purposely limited the framerate so that it should be usable with or without GPU (MTCNN requires a lot of computation).
Sometimes OpenCV needs to be reinstalled (because it can't find a given function used for image transformation), if this happens you can try to reinstall it:
(venv) >> pipenv uninstall opencv-python
(venv) >> pipenv install opencv-python >> docker build -t fer2013-prediction .>> docker run -it -p 5000:5000 fer2013-prediction:latestStop with CTRL+C
Then one can run the same test steps as before... (open input url or run test_image_local.py)
I pushed a copy of my docker image on the Docker-hub, so one can pull it:
>> docker pull valkea/fer2013-prediction:latestBut this command is optionnal, as running it (see below) will pull it if required.
Then the command to start the docker is almost similar to the previous one:
>> docker run -it -p 5000:5000 valkea/fer2013-prediction:latestStop with CTRL+C
And once again, one can run the same test steps explained above... (open input url or run test_image_local.py)
In order to create a new model .h5 file and a new weights .hdf5 file, one can use the following command:
>> python model_training.pyThis will use the default input and out names. But this can be changed using the -s (--source), -d (--destination), -b (--batch_size), -e (--epochs), -se (--steps_per_epoch), -vs (--validation_steps) parameters.
>> python model_training.py -s IN_NAME -d OUT_NAME -b 64 -e 100 -se 100 -vs 50Then these files can be converted to a .tflite file using the appropriate script:
>> python model_convert_to_TFLite.py MODEL.h5 WEIGHTS.hdf5In order to deploy this project, I decided to use Heroku.
Here is a great ressource to help deploying projects on Heroku: https://github.com/nindate/ml-zoomcamp-exercises/blob/main/how-to-use-heroku.md
So if you don't already have an account, you need to create one and to follow the process explained here: https://devcenter.heroku.com/articles/heroku-cli
Once the Heroku CLI is configured, one can login and create a project using the following commands (or their website):
>> heroku login
>> heroku create ml-fer2013Then, the project can be compiled, published and ran on Heroku, with:
>> heroku container:login
>> heroku container:push web -a ml-fer2013
>> heroku container:release web -a ml-fer2013>> python test_from_string.py --url https://ml-fer2013.herokuapp.com/predictThis should return an "Angry" label for the given face.
>> python test_from_image.py medias/Angry.jpg --url https://ml-fer2013.herokuapp.com/predict
>> python test_from_image.py medias/Fear.jpg --url https://ml-fer2013.herokuapp.com/predict
>> python test_from_image.py medias/Sad.jpg --url https://ml-fer2013.herokuapp.com/predict
>> python test_from_image.py medias/Happy.jpg --url https://ml-fer2013.herokuapp.com/predict
>> python test_from_image.py medias/Neutral.jpg --url https://ml-fer2013.herokuapp.com/predictFinally, one can send check the logs using:
>> heroku logs --tail --app ml-fer2013If you don't need this project anymore, recall to delete the Pipenv environnement before deleting the folder (otherwise it will remain on your computer).
(venv) >> exit
>> pipenv --rm