[Core Feature][flytekit] Ability to track resource utilization and gpu utilization of a python task as a plugin

[kumare3]

Motivation: Why do you think this is important?
When users are running tasks that use GPU's, CPUs and Memory, it might be useful to allow some meta-outputs that render this information

Goal: What should the final outcome look like, ideally?
There are many libraries like https://github.com/gpuopenanalytics/pynvml etc that can be used to collect stats for various resources and we could either stream or outputs these stats as a rendered PNG at the end of the run. The rendering can be done either in python or on the server and the outputs should be meta outputs.

Describe alternatives you've considered
Use other external libraries like Wandb etc

[Optional] Propose: Link/Inline OR Additional context
Write a new plugin flytekit-resource-monitoring that can load components that can output certain meta outputs.
For this, we will need support for meta-outputs.

[kumare3]

cc @EngHabu / @cosmicBboy / @eapolinario
This will be extremely useful in debugging ML jobs. We could use the same technique later to render loss functions etc for a job

[eapolinario]

Just to be clear, we could separate the work to enable meta-outputs from the Logical types work in order to work on this new flytekit plugin, right, @kumare3 ?

[kumare3]

Yup, I think we need meta outputs first and then this if at all

[github-actions]

Hello 👋, This issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will close the issue if we detect no activity in the next 7 days. Thank you for your contribution and understanding! 🙏

[github-actions]

Hello 👋, This issue has been inactive for over 9 months and hasn't received any updates since it was marked as stale. We'll be closing this issue for now, but if you believe this issue is still relevant, please feel free to reopen it. Thank you for your contribution and understanding! 🙏

[github-actions]

Hello 👋, this issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will engage on it to decide if it is still applicable.
Thank you for your contribution and understanding! 🙏