Support/integration with platforms like SageMaker, Databricks, etc.

[dberenbaum]

Report

Starting with SageMaker and Databricks, document how to work with DVC on those platforms. Need to consider how to structure this inside our docs. Some examples are listed below for reference.

SageMaker

• https://docs.wandb.ai/guides/integrations/sagemaker
• https://docs.neptune.ai/tools/sagemaker/
• https://www.comet.com/docs/v2/integrations/third-party-tools/sagemaker/
• https://wandb.ai/wandb/sm-pytorch-mnist-new/reports/Using-AWS-Sagemaker-and-Weights-Biases-Together-on-Digit-Recognition-with-MNIST---Vmlldzo4MTk3Nzg
• https://aws.amazon.com/blogs/machine-learning/track-your-ml-experiments-end-to-end-with-data-version-control-and-amazon-sagemaker-experiments/

Databricks

• https://docs.wandb.ai/guides/integrations/databricks
• https://wandb.ai/kenlee/databricks_demo/reports/Getting-Started-with-Weights-and-Biases-in-Databricks--VmlldzoyMzQyNzAx
• https://aws.amazon.com/blogs/machine-learning/managing-your-machine-learning-lifecycle-with-mlflow-and-amazon-sagemaker/
• https://heartbeat.comet.ml/integrating-comet-and-azure-databricks-4ec97703a2fe

[dberenbaum]

For SageMaker, #4756 is a start. Additional areas to address:

• Model deployment #4789
• Other data access patterns (related to https://docs.aws.amazon.com/sagemaker/latest/dg/model-access-training-data.html)
• Caching s3 outputs (requires cloud versioning: external outputs dvc#8411)
• Logging metrics in training jobs (related to remote/no-git logging dvclive#638)

[efiop]

So now that we have some basic entries for sagemaker and databricks, are there any other popular platforms that we hear constant questions about and that would benefit from us having at least some basic integration docs?

[dberenbaum]

Although there is a lot more we can do here, let's call this complete and open up new issues for future work.