You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Task resources are not correctly assigned when supplied to Dask tasks.
This seems to be caused by this PR. Basically, we were setting the resource overrides on the container to the containers resources, which is what made this approach work. However, now the resource overrides are completely empty (unless populated when this task is called from a workflow with .with_overrides(...) of course, so this is overriding the platform resources with an empty set.
Resource requests and limits should be correctly applied.
Additional context to reproduce
The following Dask task:
@task(
task_config=Dask(
workers=WorkerGroup(
number_of_workers=10,
),
),
cache_version="1",
cache=True,
)
def hello_dask_2(size: int) -> float:
# Dask will implicitly create a Client in the background by calling Client(). When executing
# remotely, this Client() will use the deployed ``dask`` cluster.
array = da.random.random(size)
return float(array.mean().compute())
results in empty requests and resources.
Screenshots
No response
Are you sure this issue hasn't been raised already?
Yes
Have you read the Code of Conduct?
Yes
The text was updated successfully, but these errors were encountered:
hamersaw
added
bug
Something isn't working
untriaged
This issues has not yet been looked at by the Maintainers
and removed
untriaged
This issues has not yet been looked at by the Maintainers
labels
May 17, 2023
Describe the bug
Task resources are not correctly assigned when supplied to Dask tasks.
This seems to be caused by this PR. Basically, we were setting the resource overrides on the container to the containers resources, which is what made this approach work. However, now the resource overrides are completely empty (unless populated when this task is called from a workflow with
.with_overrides(...)
of course, so this is overriding the platform resources with an empty set.I think we can fix this by updating the defaultResources here to be
defaultContainerSpec.Resources
like is now done in the container helper code. Then we update this line to use ResourceCustomizationModeMergeExistingResources which will merge the container resources with overrides and then apply the defaults if none are set.Expected behavior
Resource requests and limits should be correctly applied.
Additional context to reproduce
The following Dask task:
results in empty requests and resources.
Screenshots
No response
Are you sure this issue hasn't been raised already?
Have you read the Code of Conduct?
The text was updated successfully, but these errors were encountered: