Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

flux-resource list processing is very slow #5819

Closed
grondo opened this issue Mar 22, 2024 · 1 comment
Closed

flux-resource list processing is very slow #5819

grondo opened this issue Mar 22, 2024 · 1 comment

Comments

@grondo
Copy link
Contributor

grondo commented Mar 22, 2024

As noted in flux-framework/flux-sched#1137 flux resource list is very slow at processing large amounts of resources:

With ~10K nodes and ~64 cores per node the command takes 6-7s to process a sched.resource-status payload.

Profiling shows the majority of the time is spent in resources_uniq_lines(), which currently iterates over every rank in the resources set and applies the requested format to collapse lines that are identical besides the hostlist. It should be fairly simple to come up with something that's several orders of magnitude more efficient than the current approach.

@grondo
Copy link
Contributor Author

grondo commented Mar 25, 2024

I think the flux resource list slow processing has been addressed for now by #5823 and #5824

@grondo grondo closed this as completed Mar 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant