A Pytorch Implementation: Multimodal Recurrent Model with Attention for Automated Radiology Report Generation
This repository reimplements the recurrent-conv model in 2018 MICCAI paper: Multimodal Recurrent Model with Attention for Automated Radiology Report Generation [1].
The source code is licensed under CC BY 4.0 license. The contents of this repository are released under an MIT license.
The required Python packages are listed in requirements.txt
Download Indiana University Chest X-Ray dataset [2] : Original source, it is under the license.
I selected all frontal images both with impression and findings: Download link (646 MB),
After downloading, please unzip it into "IUdata" folder.
- you can train directly, the performance will be tested after each epoch.
$ python trainer.py
Before testing, please only keep one set of weights in the "model_weights" folder, e.g., 1-finding_decoder-9.ckpt, 1-image_encoder-9.ckpt, 1-impression_decoder-9.ckpt. Only three .ckpt files are allowed in model_weights folder.
Run
$ python tester.py
| BLEU_1 | BLEU_2 | BLEU_3 | BLEU_4 | METEOR | ROUGE | |
|---|---|---|---|---|---|---|
| OrignalPaper-Recurrent- Conv [1] | 0.416 | 0.298 | 0.217 | 0.163 | 0.227 | 0.309 |
| Ours-Recurrent- Conv | 0.444 | 0.315 | 0.224 | 0.162 | 0.189 | 0.364 |
If you use codes in this repository, please cite this github website address.
[1] Xue, Y., Xu, T., Long, L.R., Xue, Z., Antani, S., Thoma, G.R., Huang, X.: Multimodal recurrent model with attention for automated radiology report generation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 457–466. Springer (2018)
[2] Demner-Fushman, D., Kohli, M.D., Rosenman, M.B., Shooshan, S.E., Rodriguez, L., Antani, S., Thoma, G.R., McDonald, C.J.: Preparing a collection of radiology examinations for distribution and retrieval. J. Am. Med. Inform. Assoc. 23(2), 304–310 (2015)