Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Dask Plugin Resource Assignment Regression #3694

Closed
2 tasks done
hamersaw opened this issue May 17, 2023 · 0 comments · Fixed by flyteorg/flyteplugins#351 or flyteorg/flytepropeller#566
Closed
2 tasks done
Assignees
Labels
bug Something isn't working
Milestone

Comments

@hamersaw
Copy link
Contributor

Describe the bug

Task resources are not correctly assigned when supplied to Dask tasks.

This seems to be caused by this PR. Basically, we were setting the resource overrides on the container to the containers resources, which is what made this approach work. However, now the resource overrides are completely empty (unless populated when this task is called from a workflow with .with_overrides(...) of course, so this is overriding the platform resources with an empty set.

I think we can fix this by updating the defaultResources here to be defaultContainerSpec.Resources like is now done in the container helper code. Then we update this line to use ResourceCustomizationModeMergeExistingResources which will merge the container resources with overrides and then apply the defaults if none are set.

Expected behavior

Resource requests and limits should be correctly applied.

Additional context to reproduce

The following Dask task:

@task(
    task_config=Dask(
        workers=WorkerGroup(
            number_of_workers=10,
        ),
    ),
    cache_version="1",
    cache=True,
)
def hello_dask_2(size: int) -> float:
    # Dask will implicitly create a Client in the background by calling Client(). When executing
    # remotely, this Client() will use the deployed ``dask`` cluster.

    array = da.random.random(size)
    return float(array.mean().compute())

results in empty requests and resources.

Screenshots

No response

Are you sure this issue hasn't been raised already?

  • Yes

Have you read the Code of Conduct?

  • Yes
@hamersaw hamersaw added bug Something isn't working untriaged This issues has not yet been looked at by the Maintainers and removed untriaged This issues has not yet been looked at by the Maintainers labels May 17, 2023
@hamersaw hamersaw added this to the 1.7.0 milestone May 17, 2023
@hamersaw hamersaw self-assigned this May 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
1 participant