Add internal CLI to generate instance descriptions for CSPs

[cindyyuanjiang]

Fixes #1123.

This PR is first step to remove dependency on CSP CLIs. Ideally after we added this CLI, we can generate instance description json files for each platform and store them as resources in tools repo. Then we can add logic to use these instance description files when running tools.

Changes

Added an internal CLI spark_rapids_dev generate_instance_description [options].

Options:

• platform: currently accepts emr, dataproc and databricks-azure. Databricks-aws platform can reuse the file generated by emr, and dataproc-gke can reuse the one for dataproc.
• output_folder: directory path for the output file.

The generated json file has the following format (which is inspired by EMR CLI output):

{
  "instance_name": {
     "VCpuCount": xxx,
     "MemoryInMB": xxx,
     "GpuInfo": [
       {
         "Name": "xxx",
         "Count": xxx
       }
     ]
   },
  ...
}

For CPU instance, the entry will not have "GpuInfo".

Example json file entry for EMR platform

  "g5.4xlarge": {
    "VCpuCount": 16,
    "MemoryInMB": 65536,
    "GpuInfo":  [
      {
        "Name": "A10G",
        "Count": 1
      }
    ]
  }

Testing

spark_rapids_dev generate_instance_description --platform emr/dataproc/databricks-azure

[cindyyuanjiang]

I want to discuss: is it a good approach to add a new spark_rapids_dev CLI or should we keep it under spark_rapids?

[mattahrens]

I would rather keep it under the spark_rapids CLI instead of adding a new one. Is there a way to add the command without exposing it to users since it should be internal?

[tgravescs]

what would be the goal of keeping it under the same cli an end user would use without having a useful info message for anyone including dev to see?

[amahussein]

I would rather keep it under the spark_rapids CLI instead of adding a new one. Is there a way to add the command without exposing it to users since it should be internal?

@mattahrens
In our CLI, it is not possible to hide a cmd/argument.

[amahussein]

I want to discuss: is it a good approach to add a new spark_rapids_dev CLI or should we keep it under spark_rapids?

Thanks @cindyyuanjiang !
I will take a look at the changes.

[amahussein]

Thanks @cindyyuanjiang !
Overall, this looks going in the right direction.
I may need to take a deeper look to see if we can use more implementations from the spark_rapids_tools package.

[mattahrens]

I would rather keep it under the spark_rapids CLI instead of adding a new one. Is there a way to add the command without exposing it to users since it should be internal?

@mattahrens In our CLI, it is not possible to hide a cmd/argument.

OK, then we can have a dev CLI then for separation.

[parthosa]

Thanks @cindyyuanjiang.

Should we use a class to encapsulate the instance description structure instead of using hardcoded dictionary keys? This could improve maintainability across multiple files.

{
    "MemoryInfo": {
      "SizeInMiB": 49152
    },
    "GpuInfo": {
      "GPUs": [
        {
          "Name": "NVIDIA",
          "Manufacturer": "NVIDIA",
          "Count": 1,
          "MemoryInfo": {}
        }
      ]
    },
    "VCpuInfo": {
      "DefaultVCpus": 12
    }
}

[cindyyuanjiang]

Added Gpu info for n1-standard series for Dataproc. New entry looks like:

"n1-standard-16": {
    "VCpuCount": 16,
    "MemoryInMB": 61440,
    "GpuInfo": [
      {
        "Name": "T4",
        "Count": [
          1,
          2,
          4
        ]
      },
      {
        "Name": "P4",
        "Count": [
          1,
          2,
          4
        ]
      },
      {
        "Name": "V100",
        "Count": [
          2,
          4,
          8
        ]
      },
      {
        "Name": "P100",
        "Count": [
          1,
          2,
          4
        ]
      }
    ]
  }

cc: @tgravescs @amahussein

[parthosa]

Currently n1-standard have array type for GpuInfo.Count whereas it is an int for others. Do we want to keep this separate for any special handling for n1-standard?

Example,

"n1-standard-64": {
    "VCpuCount": 64,
    "MemoryInMB": 245760,
    "GpuInfo": [
      {
        "Name": "T4",
        "Count": [
          4
        ]
      },
      {
        "Name": "P4",
        "Count": [
          4
        ]
      }
    ]
  },

vs

  "g2-standard-4": {
  "VCpuCount": 4,
  "MemoryInMB": 16384,
  "GpuInfo": [
    {
      "Name": "L4",
      "Count": 1
    }
  ]
},

[cindyyuanjiang]

Thanks @parthosa! I have updated the GPU Count field to a list for consistency across instances and platforms.

[amahussein]

Thanks @cindyyuanjiang !
LGTME

[parthosa]

Thanks @cindyyuanjiang. LGTM

[tgravescs]

I think this comment needs updated to be array

[cindyyuanjiang]

thanks, updated.

[tgravescs]

minor nit on the comment, otherwise looks great. Thanks Cindy!