cli-upload

[georgosgeorgos]

cli-upload

Add upload functionality to the command line. It gives the user the possibility to upload specific artifacts on a server.

Given a specific version for an algorithm:

• check if that version is already on the server:
  - check if the folder bucket/algorithm_type/algorithm_name/algorithm_application/version/ exists.
• If yes, tell the user and stop the upload.
• If not, upload all the files in that version.

cli-upload relies on minio and has been tested locally using docker-compose.
cli-upload can be used to upload on a cloud or local server.

---

How to use cli-upload

Following the example in the README (in the Saving a trained algorithm for inference via the CLI command section) and assuming a trained model in /tmp/test_cli_upload, run:

gt4sd-upload --training_pipeline_name paccmann-vae-trainer --model_path /tmp/test_cli_upload --training_name fast-example --target_version fast-example-v0 --algorithm_application PaccMannGPGenerator

[cla-bot]

Thanks a lot for working on GT4SD, we strongly value contributions from our users. It appears one of the commiters in the PR did not sign the CLA for contributors. To do so, you can open an issue to sign the CLA clicking here. More details can be found here.

[cla-bot]

Thanks a lot for working on GT4SD, we strongly value contributions from our users. It appears one of the commiters in the PR did not sign the CLA for contributors. To do so, you can open an issue to sign the CLA clicking here. More details can be found here.

[drugilsberg]

@georgosgeorgos amazing work, see my comments, mostly minor changes/refactor and points of discussion.

[cla-bot]

Thanks a lot for working on GT4SD, we strongly value contributions from our users. It appears one of the commiters in the PR did not sign the CLA for contributors. To do so, you can open an issue to sign the CLA clicking here. More details can be found here.

[georgosgeorgos]

For the moment we don't have a direct way (for example os.environ) to use environment variables in configuration.py

[drugilsberg]

For the moment we don't have a direct way (for example os.environ) to use environment variables in configuration.py

The BaseSettings from pedantic are directly initialized from the env variables (where the upper case env var is read into the corresponding lower case class attribute).
To access those simply add the the to the configuration object and instantiate it. I hope this helps.

[drugilsberg]

We are getting there :)

[georgosgeorgos]

The BaseSettings from pedantic are directly initialized from the env variables (where the upper case env var is read into the corresponding lower case class attribute). To access those simply add the the to the configuration object and instantiate it. I hope this helps.

Fixed it. Thanks.

When using upload functionality locally, the server can be http and we need gt4sd_s3_secure=False in configuration.py.
I tried setting an environment variable GT4SD_S3_SECURE="False" and then use gt4sd_s3_secure: bool = distutils.util.strtobool("True") as default in configuration.py.
This works, but raises an error in one of the checks and I removed it. So at the moment the user has to manually change the value in configuration.py

[drugilsberg]

The BaseSettings from pedantic are directly initialized from the env variables (where the upper case env var is read into the corresponding lower case class attribute). To access those simply add the the to the configuration object and instantiate it. I hope this helps.

Fixed it. Thanks.

When using upload functionality locally, the server can be http and we need gt4sd_s3_secure=False in configuration.py. I tried setting an environment variable GT4SD_S3_SECURE="False" and then use gt4sd_s3_secure: bool = distutils.util.strtobool("True") as default in configuration.py. This works, but raises an error in one of the checks and I removed it. So at the moment the user has to manually change the value in configuration.py

in principle the strtobool is not needed, you can simply set gt4sd_s3_secure: bool = True and define export GT4SD_S3_SECURE-false and it should work right away.

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: GitHub Action.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email [email protected]
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: GitHub Action.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email [email protected]
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: GitHub Action.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email [email protected]
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: GitHub Action.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email [email protected]
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

[drugilsberg]

Last bit, then it's good to go

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: GitHub Action.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email [email protected]
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

[georgosgeorgos]

There is a problem with GitHub Action related to this issue - actions/runner#667. Thanks @jannisborn for finding it.

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: GitHub Action.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email [email protected]
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

[drugilsberg]

Added a minor rephrasing in the docs. Good to go, besides the CLA issue.

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: GitHub Action.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email [email protected]
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails