-
Notifications
You must be signed in to change notification settings - Fork 641
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] Task container terminate with error OOMKilled with exit code 137 #4975
Comments
🤖 Hey @spacepirate0001, great to see you back! Ran into another quirky adventure, I see? Let's dive into it. Thank you for providing detailed information about the issue you're facing. It seems like the
To diagnose and fix the issue, you should:
I also found a similar solved issue [Docs] Document how to change default platform conf when setting resources for task which might be helpful. If you're trying to override the resources allocated to tasks at runtime, you can use the
Here's the relevant code snippet from the context: @workflow
def my_pipeline(x: typing.List[int]) -> int:
return square_1(x=count_unique_numbers_1(x=x)).with_overrides(limits=Resources(cpu="6", mem="500Mi")) In this example, the I hope this helps! If you have any more questions or need further clarification, feel free to ask. Sources
|
@spacepirate0001 , how did you update the values in the task resources? |
I first tried to add it to
It did not work then I added it as
This partially worked as new task_resource run with the following:
My tasks need more resources which end up with the error OOMKilled with exit code 137 |
@spacepirate0001 , how are you updating the values? Can you share the commands you used? Also, can you confirm which |
Code is deployed via terraform module updates and I can see the changes I make being reflected on the manifest. You should try the same in your setup and see that the values don’t change beyond what I’ve mentioned. Finally I’m running flyte-binary chart in which I did not find values for task_resources at all. |
In my configuration:
inline:
task_resources:
defaults:
cpu: 500m
memory: 500Mi
storage: 500Mi
limits:
cpu: "10"
memory: 20Gi However when I try to register a workflow with tasks that has a task with a limit set to
If I run EDIT my issue was that I had previously set task-resource-attribute via |
Describe the bug
Using
inline
to override Flyte configuration fortask_resources
is not working as expected. I added the following configuration to extend value.yaml in flyte-binary chart:The added configuration did not yield the expected change on
task_resources
and task containers yaml showed the following in terms of resource settings:Expected behavior
task container resources should reflect the new configurations added to values.yaml i.e.
should have task container run with the following resources:
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
Have you read the Code of Conduct?
The text was updated successfully, but these errors were encountered: