Learn an Effective Lip Reading Model without Pains

Content

• Introduction
• Benchmark
• Dataset Preparation
• Pretrain Weights
• How to test
• How to train
• Dependencies
• Citation
• License

Introduction

This is the repository of An Efficient Software for Building Lip Reading Models Without Pains. In this repository, we provide a deep lip reading pipeline as well as pre-trained models and training settings. We evaluate our pipeline on LRW Dataset and LRW1000 Dataset. We obtain 88.4% and 56.0% on LRW and LRW-1000, respectively. The results are comparable and even surpass current state-of-the-art results. Especially, we reach the current state-of-the-art result (56.0%) on LRW-1000 Dataset.

Benchmark

Year	Method	LRW	LRW-1000
2017	Chung et al.	61.1%	25.7%
2017	Stafylakis et al.	83.5%	38.2%
2018	Stafylakis et al.	88.8%	-
2019	Yang et at.	-	38.19%
2019	Wang et al.	83.3%	36.9%
2019	Weng et al.	84.1%	-
2020	Luo et al.	83.5%	38.7%
2020	Zhao et al.	84.4%	38.7%
2020	Zhang et al.	85.0%	45.2%
2020	Martinez et al.	85.3%	41.4%
2020	Ma et al.	87.7%	43.2%
2020	ResNet18 + BiGRU (Baseline + Cosine LR)	85.0%	47.1%
2020	ResNet18 + BiGRU (Baseline with word boundary + Cosine LR)	87.5%	55.0%
2020	Our Method	86.2%	48.3%
2020	Our Method (with word boundary)	88.4%	56.0%

Dataset Preparation

1. Download LRW Dataset and LRW1000 Dataset and link lrw_mp4 and LRW1000_Public in the root of this repository:

ln -s PATH_TO_DATA/lrw_mp4 .
ln -s PATH_TO_DATA/LRW1000_Public .

2. Run scripts/prepare_lrw.py and scripts/prepare_lrw1000.py to generate training samples of LRW and LRW-1000 Dataset respectively:

python scripts/prepare_lrw.py
python scripts/prepare_lrw1000.py 

The mouth videos, labels, and word boundary information will be saved in the .pkl format. We pack image sequence as jpeg format into our .pkl files and decoding via PyTurboJPEG. If you want to use your own dataset, you may need to modify the utils/dataset.py file.

Pretrain Weights

We provide pretrained weight on LRW/LRW-1000 dataset for evaluation. For smaller datasets, the pretrained weights can be provide a good start point for feature extraction, finetuning, and so on.

Link of pretrained weights: Baidu Yun (code: ivgl)

If you can not access to provided links, please email [email protected] or [email protected].

How to test

To test our provided weights, you should download weights and place them in the root of this repository.

For example, to test baseline on LRW Dataset:

python main_visual.py \
    --gpus='0'  \
    --lr=0.0 \
    --batch_size=128 \
    --num_workers=8 \
    --max_epoch=120 \
    --test=True \
    --save_prefix='checkpoints/lrw-baseline/' \
    --n_class=500 \
    --dataset='lrw' \
    --border=False \
    --mixup=False \
    --label_smooth=False \
    --se=False \
    --weights='checkpoints/lrw-cosine-lr-acc-0.85080.pt'

To test our model in LRW-1000 Dataset:

python main_visual.py \
    --gpus='0'  \
    --lr=0.0 \
    --batch_size=128 \
    --num_workers=8 \
    --max_epoch=120 \
    --test=True \
    --save_prefix='checkpoints/lrw-1000-final/' \
    --n_class=1000 \
    --dataset='lrw1000' \
    --border=True \
    --mixup=False \
    --label_smooth=False \
    --se=True \
    --weights='checkpoints/lrw1000-border-se-mixup-label-smooth-cosine-lr-wd-1e-4-acc-0.56023.pt'

How to train

For example, to train lrw baseline:

python main_visual.py \
    --gpus='0,1,2,3'  \
    --lr=3e-4 \
    --batch_size=400 \
    --num_workers=8 \
    --max_epoch=120 \
    --test=False \
    --save_prefix='checkpoints/lrw-baseline/' \
    --n_class=500 \
    --dataset='lrw' \
    --border=False \
    --mixup=False \
    --label_smooth=False \
    --se=False  

Optional arguments:

• gpus: the GPU id used for training
• lr: learning rate. By default, we automatically applied the Linear Scale Rule in code (e.g., lr=3e-4 for 4 GPUs x 32 video/gpu and lr=1.2e-3 for 8 GPUs x 128 video/gpu). We recommend lr=3e-4 for 32 video/gpu when training from scratch. You need to modify the learning rate based on your setting.
• batch_size: batch size
• num_workers: the number of processes used for data loading
• max_epoch: the maximum epochs in training
• test: The test mode. When using this mode, the program will only test once and exit.
• weights(optional): The path of pre-trained weight. If this option is specified, the model will load the pre-trained weights by the given location.
• save_prefix: the save prefix of model parameters
• n_class: the number of total word classes
• dataset: the dataset used for training and testing, only lrw and lrw1000 are supported.
• border: use word boundary indicated variable for training and testing
• mixup: use mixup in training
• label_smooth: use label_smooth in training
• se: use se module in ResNet-18

More training details and setting can be found in our paper. We plan to include more pretrained models in the future.

Dependencies

• PyTorch 1.6
• opencv-python
• TurboJPEG and PyTurboJPEG

Citation

If you find this code useful in your research, please consider to cite the following papers:

@inproceedings{feng2021efficient,
  title={An Efficient Software for Building LIP Reading Models Without Pains},
  author={Feng, Dalu and Yang, Shuang and Shan, Shiguang},
  booktitle={2021 IEEE International Conference on Multimedia \& Expo Workshops (ICMEW)},
  pages={1--2},
  year={2021},
  organization={IEEE}
}
@article{feng2020learn,
  author       = "Feng, Dalu and Yang, Shuang and Shan, Shiguang and Chen, Xilin",
  title        = "Learn an Effective Lip Reading Model without Pains",
  journal      = "arXiv preprint arXiv:2011.07557",
  year         = "2020",
}

License

The MIT License