Estimator Output Path cannot accept Execution Input placeholder

[lgimyuen]

The arguments of sagemaker.estimator.Estimator() and sagemaker.tuner.HyperparameterTuner() doesn't accept ExecutionInput or StepInput as placeholder. E.g. for the output_path of sagemaker.estimator.Estimator() has to be fixed in the definition of workflow.

The current way around this is not to make use of stepfunctions.steps.sagemaker.TuningStep and use the generic step functions.steps.Task to invoke the arn:aws:states:::sagemaker:createHyperParameterTuningJob.sync with parameters from ExecutionInput and StepInput

[jackmuskopf]

Same issue with stepfunctions.steps.ProcessingStep

[jackmuskopf]

For instance, how do we do something like this using the stepfunctions.steps.ProcessingStep class? Specifically, S3Uri.$

"ProcessingOutputConfig": {
      "Outputs": [
        {
          "OutputName": "CleanData",
          "S3Output": {
            "S3Uri.$": "States.Format('s3://my-bucket/my/data/path/clean/{}', $$.Execution.Input['ExecutionName'])",
            "LocalPath": "/opt/ml/processing/output",
            "S3UploadMode": "EndOfJob"
          }
        }
      ]
    }

[ca-nguyen]

Hi @jackmuskopf, thank you for your feature request!
Using placeholders with Sagemaker steps has been a popular requested feature - will start work on making TuningStep Placeholder compatible shortly.

Same issue with stepfunctions.steps.ProcessingStep

Support to use placeholders with ProcessingStep has been merged and will be added with the next release (see commit)

The DataScience SDK does not currently support StepFunctions Intrinsic Functions, but with this change, it will possible to configure any property in the parameters arg with Placeholders.

For example:

parameters = {
   "ProcessingOutputConfig": {
      "Outputs": [
        {
          "OutputName": "CleanData",
          "S3Output": {
            "S3Uri": execution_input['s3_uri'],
            "LocalPath": "/opt/ml/processing/output",
            "S3UploadMode": "EndOfJob"
          }
        }
      ]
    }
}

[ca-nguyen]

Hi @jackmuskopf,

Support for placeholders for Tuning was added with #173.
It will be possible to use ExecutionInput or StepInput to define the values dynamically for any property defined in the parameter arg (see CreateHyperParameterTuningJob for the schema to follow)

Changes will be included in the the next release