Sagemaker plugin: Error handling for custom training job

[bnsblue]

[x] Capture and output subprocess stdout and stderr, and check subprocess execution return code flyteorg/flytekit#185

[ ] Write the content of errors.pb to /opt/ml/failure
Background: https://docs.google.com/document/d/118nUo2zbeiKbLnYo7A3bCyIzRuhG-cxqV3AWCJrCfYU/edit#heading=h.xg7j6wu6pw9a

[kumare3]

@bnsblue can you please help us understand what the status of this issue it?

[bnsblue]

@kumare3 There are two parts in this item: capturing the output of the subprocess and outputing the content of the errors.pb to /opt/ml/failure.

I've already done capturing the output of the subprocess, which allows Flyte itself to always be able to capture the messages from the pyflyte-execute even with the presence of the middle layer flyte_sagemaker_runner.py. However, my experience was that SM itself does not always know whether the training succeeded or not if /opt/ml/failure is empty. Outputting the content of errors.pb to /opt/ml/failure is just a measure to let SageMaker display that the training job fails in case the SM doesn't capture the true failure state.

Due to the fact that this doesn't affect the status shown on Flyte's interface, this item was never prioritized nor implemented.

I think it would require a bit work to discuss and think through what experience Flyte really wants the user to have. Once it is thought through, I believe it wouldn't require too much engineering effort to make it work. Unfortunately I don't have much chance to give it a deep thinking at the moment so I just unassigned myself. Please feel free to assign to anyone who would like to take any step.

[github-actions]

Hello 👋, This issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will close the issue if we detect no activity in the next 7 days. Thank you for your contribution and understanding! 🙏

[github-actions]

Hello 👋, This issue has been inactive for over 9 months and hasn't received any updates since it was marked as stale. We'll be closing this issue for now, but if you believe this issue is still relevant, please feel free to reopen it. Thank you for your contribution and understanding! 🙏

[github-actions]

Hello 👋, this issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will engage on it to decide if it is still applicable.
Thank you for your contribution and understanding! 🙏