Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Function jobs shouldn't request resources by default #14440

Closed
dbadura opened this issue May 27, 2022 · 4 comments
Closed

Function jobs shouldn't request resources by default #14440

dbadura opened this issue May 27, 2022 · 4 comments
Assignees
Labels
area/serverless Issues or PRs related to serverless kind/bug Categorizes issue or PR as related to a bug.
Milestone

Comments

@dbadura
Copy link
Contributor

dbadura commented May 27, 2022

Description
In current implementation by default we enforce build profile.

Expected result
By default the build profiles are not applied.

Actual result
Having those values set as default, that:

  • function deployment request 500m cpu
  • function build job request 1000m cpu
    let's consider following situation:
    User creates 3 function on node with 2000cpu available:
  • first job requires 1000m, the available left is 1000m
  • job finished and then we deploy the function which requires 500m, the available cpu is 1500m
  • user deploy second function and at the end available cpu is 1000m
  • user deploy third function. The job can fit, because we have 1000m cpu available and at the end we deploy function and cluster have available 500m cpu.
  • the user want to upgrade one of those three functions and the upgrade stuck, because the job doesn't have enough space.

By default we shouldn't apply any build profiles or at least not enforce minimum request for build jobs.
I would consider to also remove the default profile for functions if the user didn't specify.

Steps to reproduce

Troubleshooting
Settings for minimum requests: https://github.com/kyma-project/kyma/blob/main/components/function-controller/pkg/apis/serverless/v1alpha1/function_validation.go#L28=

@dbadura dbadura added kind/bug Categorizes issue or PR as related to a bug. area/serverless Issues or PRs related to serverless labels May 27, 2022
@kwiatekus kwiatekus added this to the 2.4 milestone May 27, 2022
@dbadura dbadura self-assigned this May 30, 2022
@dbadura dbadura assigned moelsayed and unassigned dbadura and moelsayed Jun 1, 2022
@kwiatekus
Copy link
Contributor

function build job cannot start in an over-commited CPU environment.
The following shows the scenario: CPU utilisation is really low ( ~14%) but CPU commitment is ~100%

Screenshot 2022-06-01 at 09 57 39

We should lower the overcommitment of CPU for kyma components

@dbadura
Copy link
Contributor Author

dbadura commented Jun 1, 2022

The code which check minimum values requested:

type MinFunctionResourcesValues struct {
MinRequestCpu string `envconfig:"default=10m"`
MinRequestMemory string `envconfig:"default=16Mi"`
}
type MinBuildJobResourcesValues struct {
MinRequestCpu string `envconfig:"default=200m"`
MinRequestMemory string `envconfig:"default=200Mi"`
}

Also look here in charts, for example:

minRequestCpu: "10m"
minRequestMemory: "16Mi"

@kwiatekus
Copy link
Contributor

kwiatekus commented Jun 1, 2022

We should

  • check if we can allow to set any value for requested CPU time and CPU limits for function builds (even empty)
  • analyse the actual CPU utilisation (% of the requested CPU time ) for kyma components that sit on most of the requested CPU
  • propose a PR with adjusted requested CPU time for kyma components

@kwiatekus
Copy link
Contributor

Thanks
We have reduced the cpu overcommitment across kyma components - saving cpu cycles for build jobs.
Additionally, we removed the requested cpu and memory for the build jobs by default - allowing for more flexible job scheduling in scenarios where cpu is limited on the worker nodes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/serverless Issues or PRs related to serverless kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests

3 participants