[Change Proposal] Support Torch .pt ML trained models in integration packages

[susan-shu-c]

What problem is this proposal solving?

Users should be able to easily install supervised ML models. Current, integrations support DataFrame Analytics models, as introduced in

• [Change Proposal] Support ML trained models in integration packages #216

Now we have more supported language models. They are shipped in a .pt format instead of .json configurations. Integrations will need to support the installation of these models.

• Note: Trained model API compressed_definition. The .pt models will need to be converted to base64 format to place into the integration packages.
• Note: To support NLP models, support for the add trained models' vocabulary is required as well. link

Where the solution will need to be implemented, i.e. which parts, if any, of the Elastic Stack will be impacted.

• Create PR in the https://github.com/elastic/package-spec repo to update the ml_model asset type. Possibly in this file?
  • PR -
• Create PR in https://github.com/elastic/kibana to add support (installing the asset using the existing create trained model API) for the model asset type into Fleet. Possibly the handleMlModelInstall chunk
  • PR -
• ? Create PR in https://github.com/elastic/integrations to add the model .pt containing the model and other model package assets themselves

Additional information

Currently, models are constrained to 400mb size. We've yet to require an increase, but it's worth noting in the future if we do develop larger models.

Links:

• Eland .pt: https://github.com/elastic/eland/blob/79d9a6ae29624d15840a48465fc2aa87dcbf7ace/eland/ml/pytorch/traceable_model.py#L62
• Trained model API compressed_definition. The .pt models will need to be converted to base64 format to place into the integration packages.