Investigate how distributed training can be used for custom training jobs

[bnsblue]

• compose an exemplar docker file for users who wants to use openmpi + horovod to do distributed training. (https://github.com/lyft/flyteplayground/pull/61)
• compose an horovod-based tensorflow distributed training example. (https://github.com/lyft/flyteplayground/pull/61)
• change the arguments of subprocess.run in flytekit_sagemaker_runner.py to allow the stdout and the stderr to be captured by the parent. (Capture and output subprocess stdout and stderr, and check subprocess execution return code flytekit#185)
• in the SageMaker flyteplugin, inject the sagemaker_enable_mpi=true in hyperparameters (Adding logic to insert the MPI-enabling hyperparameter in SageMaker plugin flyteplugins#124)
• in flytekit, provide distributed_training_context with wf_params (Enabling bare-bones distributed training for SageMaker and prepare for enabling distributed training for PyTorch / Tensorflow plugins flytekit#173)
• in flytekit, provide an interface that allows users to specify a predicate to decide which of the worker can persist its output; also provide a default predicate (this can apply to TF and PyTorch distributed training tasks as well)
  (Enabling bare-bones distributed training for SageMaker and prepare for enabling distributed training for PyTorch / Tensorflow plugins flytekit#173)
• Add a field in flytekit and flyteidl to allow users to specify the distributed protocol (Adding necessary fields in training job message for distributed training  flyteidl#84, Enable distributed training on SageMaker for custom training job task  flytekit#195)

[bnsblue]

using mpirun gives us more information compared to using horovodrun

[bnsblue]

We can leverage this https://github.com/aws/sagemaker-training-toolkit/blob/c999c12941408a89969984079f7ddfeddf272882/src/sagemaker_training/mpi.py#L13
in the horovod_launcher.py