[FEA] Add Estimation Model to Qualification CLI

[amahussein]

Signed-off-by: Ahmed Hussein (amahussein) [email protected]

Fixes #869

• Add estimation_model to qualification arguments
• Refactor the job sumbission to run as concurrent processes as we need to run both qualification and profiling tool in the xgboost model
• Remove --per-sql from the allowed list of qualification tool because it is used in the XGBOOST model
• Import qualx code into user_tools repo
• Running qual CLI with --estimation_model XGBOOST runs the prediction model and generate the results as intermediate output, but it won't affect the final results
• There are 2 files generated inside the final output directory qual_*_app.csv and qual_*_sql

Notes:

• The estimation model uses on-prem for now
• The remaining work will be WIP through tasks listed in issue-806
  • we want to extract the readDataFormat from the profiler output of each application
  • This probably will be a new class to hold appMetadata
  • Modify the modeling code to process one app at a time. Depending on the metadata, the prediction model will be loaded.
  • Prediction model uses information from both Profiler and Qual CSV files. So, we will need to handle errors that could be raised from applications that do not exist in both tools outputs.
  • The new Speedups are generated. Then we override the original Speedup estimation with the Qx Prediction, “estimated_df”
  • The “estimated_df” should be similar to the legacy qual DF.
  • The report generation (stdout+csv file) code won’t need to be changed.

[mattahrens]

minor preference to use lowercase for argument values: XGBOOST --> xgboost

[amahussein]

Thanks @mattahrens.

Done!
The CLI handles both lower/upper-cases. I changed the comments to lower-case which reflects on the output of the --help command.

[parthosa]

Thanks @amahussein.