Handle init_blocks in scaling strategy, rather than special-casing it #3283
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is part of issue #3278 tidying up job and block management. Now init_blocks scale out happens on the first strategy poll, not at executor start - that will often delay init_blocks scaling by one strategy poll period compared to before this PR.
benclifford
force-pushed
the
benc-scaling-init-in-strategy
branch
from
f86d403 to
8127310
Compare
|
failing a precondition in test_drain for exactly the "Changed Behaviour" reason - now that test will be reliant on strategy poll period so it can probably be made to use a smaller strategy period |
benclifford
added a commit
that referenced
this pull request
Mar 25, 2024
This will now scale in blocks using the job status poller scale in code, which means the DFK does not need to send its own BLOCK_INFO monitoring messages. minimalish change of which blocks get scaled in at shutdown - to come from the jobstatuspoller list: that will get pending blocks scaled in at shutdown, I think, but will now push the dynamically updated list to the cac hed-side of the cache poll... what does that change? we will now be delayed in s eeing ended jobs, but the executor.status data is already out of date in that se nse the moment the call returns (but *less* out of date) this patch is deliberately minimalist in that it does not attempt to move the scale down code - this is a PR about changing behaviour, not about rewriting the scale down strategy more seriously. the behaviour change is to move towards treating the jobstatuspoller pollitem status as the source of best-estimated truth. other work should probably do that moving, to complement the recent init_blocks handling PR #3283
benclifford
added a commit
that referenced
this pull request
Mar 26, 2024
This will now scale in blocks using the job status poller scale in code, which means the DFK does not need to send its own BLOCK_INFO monitoring messages. minimalish change of which blocks get scaled in at shutdown - to come from the jobstatuspoller list: that will get pending blocks scaled in at shutdown, I think, but will now push the dynamically updated list to the cac hed-side of the cache poll... what does that change? we will now be delayed in s eeing ended jobs, but the executor.status data is already out of date in that se nse the moment the call returns (but *less* out of date) this patch is deliberately minimalist in that it does not attempt to move the scale down code - this is a PR about changing behaviour, not about rewriting the scale down strategy more seriously. the behaviour change is to move towards treating the jobstatuspoller pollitem status as the source of best-estimated truth. other work should probably do that moving, to complement the recent init_blocks handling PR #3283
benclifford
added a commit
that referenced
this pull request
Mar 26, 2024
This will now scale in blocks using the job status poller scale in code, which means the DFK does not need to send its own BLOCK_INFO monitoring messages. minimalish change of which blocks get scaled in at shutdown - to come from the jobstatuspoller list: that will get pending blocks scaled in at shutdown, I think, but will now push the dynamically updated list to the cac hed-side of the cache poll... what does that change? we will now be delayed in s eeing ended jobs, but the executor.status data is already out of date in that se nse the moment the call returns (but *less* out of date) this patch is deliberately minimalist in that it does not attempt to move the scale down code - this is a PR about changing behaviour, not about rewriting the scale down strategy more seriously. the behaviour change is to move towards treating the jobstatuspoller pollitem status as the source of best-estimated truth. other work should probably do that moving, to complement the recent init_blocks handling PR #3283
benclifford
added a commit
that referenced
this pull request
Mar 27, 2024
This will now scale in blocks using the job status poller scale in code, which means the DFK does not need to send its own BLOCK_INFO monitoring messages. minimalish change of which blocks get scaled in at shutdown - to come from the jobstatuspoller list: that will get pending blocks scaled in at shutdown, I think, but will now push the dynamically updated list to the cac hed-side of the cache poll... what does that change? we will now be delayed in s eeing ended jobs, but the executor.status data is already out of date in that se nse the moment the call returns (but *less* out of date) this patch is deliberately minimalist in that it does not attempt to move the scale down code - this is a PR about changing behaviour, not about rewriting the scale down strategy more seriously. the behaviour change is to move towards treating the jobstatuspoller pollitem status as the source of best-estimated truth. other work should probably do that moving, to complement the recent init_blocks handling PR #3283
khk-globus
approved these changes
Mar 27, 2024
From an end functionality perspective, that's checking that init_blocks actually gets used to start blocks, rather than a later part of the strategy code: with strategy With the other two strategies that's less observable from the outside, I think: |
rjmello
added a commit
to globus/globus-compute
that referenced
this pull request
Jun 27, 2024
As of Parsl PR Parsl/parsl#3283, the initial HTEX block scale out occurs on the first strategy poll, not at HTEX start. Thus, our tests should use a small strategy period to speed them up and avoid timeouts.
rjmello
added a commit
to globus/globus-compute
that referenced
this pull request
Jun 28, 2024
As of Parsl PR Parsl/parsl#3283, the initial HTEX block scale out occurs on the first strategy poll, not at HTEX start. Thus, our tests should use a small strategy period to speed them up and avoid timeouts.
rjmello
added a commit
to globus/globus-compute
that referenced
this pull request
Jun 28, 2024
As of Parsl PR Parsl/parsl#3283, the initial HTEX block scale out occurs on the first strategy poll, not at HTEX start. Thus, our tests should use a small strategy period to speed them up and avoid timeouts.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is part of issue #3278 tidying up job and block management.
Changed Behaviour
Now init_blocks scale out happens on the first strategy poll, not at executor start - that will often delay init_blocks scaling by one strategy poll period compared to before this PR.
Type of change