Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add Baidu warpctc option to reproduce CTC results of our paper. #209

Merged
merged 2 commits into from
Aug 3, 2020

Conversation

ku21fan
Copy link
Contributor

@ku21fan ku21fan commented Aug 3, 2020

While we used Baidu warpctc in the paper, we use PyTorch CTC in this repository.
The main reason why we changed the Baidu warpctc into PyTorch CTC is that Baidu warpctc is not easy to use for PyTorch >= 1.2 (see here, maybe you can still install warpctc with PyTorch 1.5 from the source (see here), I did not check it though.).
(our initial codes used Baidu warpctc before PyTorch CTC was released.)

However, we found that the accuracy of PyTorch CTC apparently lower than Baidu warpctc in some results (ex. CRNN, about 1%).
And thus, reproducing the CRNN result of our paper was difficult.
This problem is related to PyTorch version and Baidu warpctc issue.
Sorry for the inconvenience.

To solve this problem,

  1. We provide a guideline to reproduce our paper by using PyTorch 1.1 and Baidu warpctc.
  2. We also upload the pretrained model CRNN-PyTorchCTC.pth (trained with the default setting of this repo which uses PyTorch CTCLoss) and report the result of this model here.

The details are below.

  1. Guideline to reproduce our paper by using PyTorch 1.1 and Baidu warpctc.
# we recommend using conda. If you don’t use conda, just ignore the first 3 lines.
conda create -n warpctc python=3.6
conda activate warpctc
conda install ipython
pip install torch==1.1
pip install warpctc-pytorch11-cuda90
pip install lmdb pillow torchvision==0.3 nltk natsort

Then run the train.py with --baiduCTC option as below.

CUDA_VISIBLE_DEVICES=0 python3 train.py \
--train_data data_lmdb_release/training --valid_data data_lmdb_release/validation --select_data MJ-ST --batch_ratio 0.5-0.5 \
--Transformation None --FeatureExtraction VGG --SequenceModeling BiLSTM --Prediction CTC \
--exp_name CRNN-baiduCTC --baiduCTC

If the current_accuracy at 2000 iter is about 48~50%, Baidu warpctc is working right.
In the case of PyTorch CTCLoss, the current_accuracy at 2000 iter is about 45~46%.

We upload the pretrained model CRNN-BaiduCTC.pth (trained with --baiduCTC option which uses Baidu warpctc) and report the result of this model here.

IIIT5k_3000: 82.300	SVT: 82.844
IC03_860: 92.674	IC03_867: 92.849
IC13_857: 90.548	IC13_1015: 88.670	
IC15_1811: 69.244	IC15_2077: 63.746	
SVTP: 70.698		CUTE80: 62.500

Total accuracy of a unified evaluation dataset (8,539 images in total) is 78.112;
3,000 from IIIT, 647 from SVT, 867 from IC03, 1015 from IC13, 2,077 from IC15, 645 from SP, and 288 from CT.
  1. We also upload the pretrained model CRNN-PyTorchCTC.pth (trained with the default setting of this repo which uses PyTorch CTCLoss) and report the result of this model here.
IIIT5k_3000: 82.567	SVT: 80.526
IC03_860: 92.791	IC03_867: 92.272
IC13_857: 89.732	IC13_1015: 88.768
IC15_1811: 66.317	IC15_2077: 61.387
SVTP: 65.736		CUTE80: 64.931

Total accuracy of a unified evaluation dataset (8,539 images in total) is 77.117;
3,000 from IIIT, 647 from SVT, 867 from IC03, 1015 from IC13, 2,077 from IC15, 645 from SP, and 288 from CT.

The results could be slightly different up to random seed, your computing environment, and etc.

Hope this model/result helps your project.

If you found some issues, please let us know.
Best.

@gwkrsrch gwkrsrch merged commit 3c2c89a into clovaai:master Aug 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants