Research: Task Execution Provider for Python

[odubajDT]

Goal

Research possibilities how to run python scripts as part of Tasks in KLT. Additionally create a PoC with a brief implementation of this functionality

[odubajDT]

Summary of the research

PoC available here

The first part of the research was about looking for a runtime image for python. Basically the best solution was to use the official python runtime image, which has regular updates and fixes. Already existing image fitting our requirements regarding the installed dependencies and executing either python script from a ConfigMap or a remote repository (via URL) was not available. Therefore the solution is to create a custom image based on the official python runtime image which runs a simple bash script, which actually runs the injected python script with the appropriate env variables set. The whole concept is pretty similar to what is currently used for a function runtime image running JS/TS scripts with deno. The pros of this approach is that we already have an existing logic for running tasks as jobs and the only adaptations needed is just to use a proper image to be run. On the other side, this approach is not 100% reusable nor extensible to other languages without actually adapting (in a small manner) the code handling the Task reconciliation logic.

Points to consider for implementation

• creating custom python-runner image with a simple bash script (with installing the dependencies described in the KEP) - first version available as part of the linked PoC
• adapting pipelines to build, test and release the custom python-runner image -> we can follow the path which is already made with functions-runtime image
• introduce new parameters (Runtime, CmdParameters) into the Task and TaskDefinition CRD and propagate these values to the actual job
• introduce a logic to create a jobs running tasks with python-runtime custom image - previous to this step I would suggest a refactoring of the job creation logic is needed -> I would introduce a JobCreator interface, which depending on the value set in the Runtime will choose the correct runtime image and use it to create and run a job. With these changes it should be easier to extend this logic in the future with the support of more runtime images (for example bash)
• document the newly supported functionality - KeptnTask, KeptnTaskDefiniton, new parameters in CRDs, new env variable in the klt-operator deployment
• create examples for users

Points already covered

• Configurable Timeout and number of Retries for the KeptnTask is already implemented in the klt-operator

Points to follow up

The KEP defines a requirement for the users to be able to return a JSON object from the user-defined script as a resulting status after finishing. Here there are multiple options how to handle this (from flushing it into the job pod logs up to storing them to a standardized file in a volume). This part was not deeply researched and for now is out of the scope.