New model upload format #17
Conversation
Format is compatible with https://docs.aws.amazon.com/deepracer/latest/developerguide/deepracer-troubleshooting-service-migration-errors.html The current best way for local training seems to be: 1. Upload model to s3 2. Import model from s3 in AWS DeepRacer console.
| cp -v $MODEL_FILE checkpoint/model/ | ||
| cp -v $METADATA_FILE checkpoint/model/ | ||
|
|
||
| CHECKPOINT_FILES=$MODEL_DIR/${CHECKPOINT}* |
There was a problem hiding this comment.
It seems tha according to the spec we need current checkpoint and previous one (so 2 checkpoint sets) - so the previous checkpoint should be copied or simply all checkpoints (there are two sets per last and best anyways), i.e. CHECKPOINT_FILES=$MODEL_DIR/*.ckpt.*
There was a problem hiding this comment.
It works for race submission with a single checkpoint. Does something break without 2 checkpoints?
If we really need multiple checkpoints then we should upload all checkpoints.
Uploading all checkpoints takes longer, so I would only upload them all if this is really needed.
| aws s3 cp $filename s3://$S3_BUCKET/$S3_PREFIX/model/ | ||
| done | ||
| # Cleanup upload destination | ||
| aws s3 rm --recursive s3://$S3_BUCKET/$S3_PREFIX/ |
There was a problem hiding this comment.
this one is removing all other folders which makes AWS unhappy, change it to: aws s3 rm s3://$S3_BUCKET/$S3_PREFIX/model --recursive
There was a problem hiding this comment.
I forgot to include the directory upload-template. So some files were missing and that was causing issues.
I would always clean up the whole prefix to ensure that only newly uploaded files are present. Otherwise, unknown old files could be affecting the import and your model.
|
|
||
| tar -czvf ${CHECKPOINT}-checkpoint.tar.gz checkpoint/* | ||
| # Upload files to s3 | ||
| aws s3 sync checkpoint/ s3://$S3_BUCKET/$S3_PREFIX/ |
There was a problem hiding this comment.
here it should be just: aws s3 sync checkpoint/model s3://$S3_BUCKET/$S3_PREFIX/model
There was a problem hiding this comment.
The goal is to always upload a complete model with all required files, not just the model files. That includes also reward_function.py and ip/hyperparameters.json.
There was a problem hiding this comment.
The tar is needed to keep an archive of what was uploaded. Once a new best model is found, sagemaker deletes the old best model so you would no longer have a copy.
Add missing directory with upload template. The template includes a generic reward function and generic hyperparameters.
Fixes #16
Model format is compatible with
https://docs.aws.amazon.com/deepracer/latest/developerguide/deepracer-troubleshooting-service-migration-errors.html
The current best way for local training seems to be: