How to get the document frequencies?

[Cppowboy]

If using some other corpus, get the document frequencies into a similar format as "coco-val-df", and put them in the data/ folder as a pickle file. Then set mode to the name of the document frequency file (without the '.p' extension).

How to get the document frequencies?

[awkrail]

I found how to calculate ngram and save a .p file here.
https://github.com/ruotianluo/self-critical.pytorch/blob/master/scripts/prepro_ngrams.py

The target dataset of this script is MSCOCO, but it may be helpful to you.