Add checksum checks for all MLPerf inference dataset and model downloads

[arjunsuresh]

We need to add checksum checks for all models and datasets used in MLPerf inference. For folder downloads, we can use checksum file like done here

• ResNet50
• DLRMv2
• Stable Diffusion
• LLAMA2
• RetinaNet
• Bert
• 3d-unet
• Mixtral
• GPTJ

[gfursin]

Hi Arjun. Just a quick question: does the recent MD5SUM check mechanism in CM supports MacOS and Windows? Do we have some tests for Windows and MacOS in GitHub actions (I think I saw somewhere that GitHub now has Windows Server support in workflows)? Thanks!

[arjunsuresh]

Hi @gfursin the code to do the check is in python and should work on Windows too. But currently we don't have any Windows specific tests for this. We do have Windows and macOS gh actions for CM installation and ABTF inference - that covers CHECKSUM check for individual files - but nothing yet for folders.

[gfursin]

Hi @gfursin the code to do the check is in python and should work on Windows too. But currently we don't have any Windows specific tests for this. We do have Windows and macOS gh actions for CM installation and ABTF inference - that covers CHECKSUM check for individual files - but nothing yet for folders.

Cool! Thank you! That's already a very good starting point!

[anandhu-eng]

Hi @arjunsuresh , does the env variabe CM_EXTRACT_EXTRACTED_CHECKSUM_FILE here and CM_DOWNLOAD_CHECKSUM_FILE here have same goal of being the path variable to checksum file for extracted files?

[arjunsuresh]

Yes

[arjunsuresh]

@anandhu-eng sorry - I was not clear in the earlier reply. `

CM_EXTRACT_EXTRACTED_CHECKSUM_FILE` is the file having the checksums for the files inside the extracted folder.

Even though mostly we download a single file, when we use rclone etc we do download a folder. CM_DOWNLOAD_CHECKSUM_FILE is the file having the checksums for the files inside such downloaded folder.

[anandhu-eng]

Closing the issue, as checksum for the listed models and their datasets have been added.