Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make rootish heuristic sensitive to size of task group #8005

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Commits on Jul 15, 2023

  1. Make rootish heuristic sensitive to size of task group

    Currently we require rootish tasks to belong to a task group that has no
    more than five dependencies.
    
    Sometimes (as in a recent Xarray workload showed to me by @dcherian)
    this doesn't work because there are a few dependencies, but still far
    fewer than there are tasks in the task group.  In his case there were
    6000 tasks in the task group and 6 dependencies of that group, and so it
     was erroneously classified as non-rootish.
    
    This PR proposes a change to the logic, where we accept tasks whose
    groups have <1% of the number of dependencies as they have tasks in the
    group.  So in Deepak's case because there are fewer than 6000 * 1% == 60
    dependencies, these tasks get classified as rootish.
    
    Future note, we're doing this dynamically now, maybe we should be
    looking at data stored instead of number of tasks.
    mrocklin committed Jul 15, 2023
    Configuration menu
    Copy the full SHA
    8aaf5e5 View commit details
    Browse the repository at this point in the history

Commits on Jul 28, 2023

  1. Configuration menu
    Copy the full SHA
    e3d393b View commit details
    Browse the repository at this point in the history