unable to define resources in jobtemplate with scheduled tests

[aachermann]

Describe the enhancement you'd like to see
We are currently experimenting with Testkube on OpenShift and have encountered an issue while attempting to define resources for tests that include the schedule parameter. Despite our attempts, we have been unable to assign a resource definition to the resulting CronJob. We have defined the resources within the executionRequest section of the test using the default Executor, k6. Can you suggest a solution for this problem?

it goes in the same direction as my colleague has already made a request. (#2943)

Additional context

[vsukhin]

hey @aachermann can you please provide a sample of what you did and what didn't work?

[aachermann]

test definition:

apiVersion: tests.testkube.io/v3
kind: Test
metadata:
  name: k6-test
spec:
  type: k6/script
  content:
    type: string
    data: "import http from 'k6/http';\nimport { check, sleep } from 'k6';\nexport const options = {\n stages: [\n   { duration: '5s', target: 20 },\n   { duration: '5s', target: 10 },\n ],\n thresholds: { // define the failure criteria here\n    \"http_req_duration\": [\"p(95)<500\"],  // 95% of requests must complete below 500ms\n    \"http_req_failed\": [\"rate<0.1\"], // max 10% of requests can fail\n    \"http_req_waiting\": ['p(95)<100'] // 95% of requests must wait less than 100ms for a response\n    //\"http_reqs\": ['rate>50'] // the number of requests per second must be greater than 50\n},\n};\nexport default function () {\n // Check the availability of the server\n let webserver = 'maskedtestwebservier/'\n const res = http.get(webserver);\n check(res, { 'status was 200': (r) => r.status == 200 });\nsleep(1);\n}"
  schedule: "*/1 * * * *"
  executionRequest:
    jobTemplate: |
      spec:
        template:
          spec:
            containers:
            - name: {{ .Name }}
              resources:
                requests:
                  cpu: 10m
                  memory: 50Mi
                limits:
                  cpu: 500m
                  memory: 1Gi

and the resulting CronJob Object:

kind: CronJob
apiVersion: batch/v1
metadata:
  name: k6-test-tests
  labels:
    testkube: tests
spec:
  schedule: '*/1 * * * *'
  concurrencyPolicy: Forbid
  suspend: false
  jobTemplate:
    metadata:
      creationTimestamp: null
    spec:
      backoffLimit: 0
      template:
        metadata:
          creationTimestamp: null
        spec:
          containers:
            - name: curlimage
              image: curlimages/curl
              command:
                - sh
                - '-c'
              args:
                - >-
                  curl -X POST -H "Content-Type: application/json" -d '{}'
                  "http://testkube-api-server:8088/v1/tests/k6-test/executions"
              resources: {}
              terminationMessagePath: /dev/termination-log
              terminationMessagePolicy: File
              imagePullPolicy: Always
          restartPolicy: Never
          terminationGracePeriodSeconds: 30
          dnsPolicy: ClusterFirst
          securityContext: {}
          schedulerName: default-scheduler
      ttlSecondsAfterFinished: 180
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1

[aachermann]

Hi @vsukhin
I think it's wrong how we define the resources for our Test-Jobs. Can you provide an example how we can pass resource definition from a test which creates a cronjob? (Scheduled Test) - if that's even possible.

[vsukhin]

Hey @aachermann
CronJob is created by Testkube Operator using this template https://github.com/kubeshop/helm-charts/blob/main/charts/testkube-operator/cronjob-template.yml
If you want to change the cron job definition, you can adjust it

[aachermann]

Hi @vsukhin
it would be great if we could customize the resources in the values ​​file. we render the helm chart and then have a large template with the defined values. This cronjob template is now stored as a base64 value in the deployment of the testkube-operator-controller-manager. this is then very ugly to patch, since everything has to be decoded with base64.

so it would be great if we don't have to build a mechanism that customize the helm template for us when there are build in tools that handles the cloud native stuff for the deployment. (values-file)

but I've tested to patch the deployment-artefact of the testkube-operator-controller-manager and it worked - it's just that it doesn't make sense for me to patch it like that.
I hope you understand how what I mean....

[vsukhin]

hey @aachermann yes, I agree. it's not too convinient, but it will require some dev efforts. let's check priorities with @TheBrunoLopes about it

[vsukhin]

@aachermann will this approach https://github.com/kubeshop/helm-charts/blob/main/charts/testkube-api/templates/configmap.yaml work for you?

[aachermann]

@vsukhin
if we can define the requests for the cronjob per test, analogous to normal test jobs, that would help us a lot.
something like this:

apiVersion: tests.testkube.io/v3
kind: Test
metadata:
  name: k6-test
spec:
  type: k6/script
  schedule: "*/2 * * * *"
  content:
    type: string
    data: k6data
  executionRequest:
    cronjobTemplate: |
      spec:
        jobTemplate:
          spec:
            template:
              spec:
                containers:
                  - name: {{ .Name }}
                    resources:
                      requests:
                        cpu: 10m
                        memory: 50Mi
                      limits:
                        cpu: 1
                        memory: 1Gi

[vsukhin]

hey @aachermann this looks pretty straightforward. let's @TheBrunoLopes to prioroitise this task

[tkonieczny]

@vsukhin I built the CLI and checked it with this example: https://github.com/kubeshop/testkube/blob/92f030ef2a96b99a7d9f1131d4827c43971ef020/docs/docs/concepts/tests/tests-creating.md#changing-the-default-cronjob-template-used-for-scheduled-test-execution - looks like it's not set

[vsukhin]

it should be set by testkube-operator, the question is do we have one from develop branch

[vsukhin]

Ran as an example in integration cluster
kubectl testkube create test --git-branch main --git-uri https://github.com/kubeshop/testkube-example-cypress-project.git --git-path "cypress" --name cj-test --type cypress/project --cronjob-template cronjob.yaml --schedule "*/5 * * * *"
kubectl get cronjobs -n testkube
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
cj-test-tests */5 * * * * False 0 32s

kubectl describe cronjobs cj-test-tests -n testkube
Name: cj-test-tests
Namespace: testkube
Labels: testkube=tests
Annotations: test: kube
Schedule: */5 * * * *