-
Notifications
You must be signed in to change notification settings - Fork 61
Adding proto definitions for supporting SageMaker TrainingJob (built-in algorithms) and HyperparameterTuningJob #66
Conversation
|
||
package flyte.plugins.sagemaker; | ||
|
||
option go_package = "github.com/kumare3/awsflyteplugins/gen/pb-go/proto"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmmm I think you should follow the regular path here... of other plugins...
option go_package = "github.com/lyft/flyteidl/gen/pb-go/flyteidl/plugins";
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since this has a lot of messages, can you actually add one extra level of nesting?
option go_package = "github.com/lyft/flyteidl/gen/pb-go/flyteidl/plugins/sagemaker";
If this works, can you also remove the SageMaker prefix of the messages below?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sure let me give it a try
HPOJobObjective Objective = 2; | ||
int64 MaxNumberOfTrainingJobs = 3; | ||
int64 MaxParallelTrainingJobs = 4; | ||
ParameterRanges ParameterRanges = 5; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this really a "config"? not an input of the job?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was trying to follow the CRD definition of the amazon-sagemaker-operator-for-k8s. But I think what you suggested here makes more sense.
ParameterRanges ParameterRanges = 5; | ||
} | ||
|
||
message SagemakerHPOJob { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I presume you meant for this to go into the custom field of the task?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes
int64 max_wait_time_in_seconds = 2; | ||
} | ||
|
||
message VpcConfig { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should leave this out. This should be a backend configuration IMO
|
||
message TrainingJob { | ||
string region = 1; | ||
string role_arn = 2; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this should be the same role that we are using for the workflow?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right. That means we do not need to define this as a part of the message?
protos/flyteidl/plugins/sagemaker/hyperparameter_tuning_job.proto
Outdated
Show resolved
Hide resolved
Lets just mostly remove the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Awesome! I think just make sure all messages and fields have docs and it's good to go!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
…in algorithms) and HyperparameterTuningJob (#66) * adding sagemaker protos Co-authored-by: Haytham AbuelFutuh <[email protected]>
TL;DR
This PR adds the essential proto messages for the support of SageMaker TrainingJob and HyperparameterTuningJob.
Type
Are all requirements met?
Complete description
This PR adds the essential proto messages for the support of SageMaker TrainingJob and HyperparameterTuning.
The major messages added include
HyperparameterTuningJob
,TrainingJob
, andParameterRanges
.The additions allow users to:
Tracking Issue
flyteorg/flyte#255
Follow-up issue
flyteorg/flyte#431