Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase default timeout of remote Executable requests #15587

Merged
merged 1 commit into from
Dec 10, 2024

Conversation

bolekk
Copy link
Contributor

@bolekk bolekk commented Dec 10, 2024

Remote Target executions are based on deltaStage parameter specified in the workflow spec.
Target capability nodes (e.g. chain writers) are invoked one-by-one with deltaStage seconds between invocations.

This timeout is the maximum total time all target invocations can take.
For example, with the old one minute setting, and a workflow using deltaStage=45sec, the invocations will look as follows:

  1. First writer called at 0:00
  2. Second writer called at 0:45
  3. Timeout at 1:00

Workflow Engine requires F+1 target executions to complete successfully. When F=5, and timeout set to 1 min, it never has a chance to do so.

Bumping to a larger value, which is closer to max workflow execution time (10 mins). Should give us plenty of headroom for DONs with F=5 and deltaStage up to 1 min.

@bolekk bolekk requested review from a team as code owners December 10, 2024 02:40
Copy link
Contributor

I see you updated files related to core. Please run pnpm changeset in the root directory to add a changeset as well as in the text include at least one of the following tags:

  • #added For any new functionality added.
  • #breaking_change For any functionality that requires manual action for the node to boot.
  • #bugfix For bug fixes.
  • #changed For any change to the existing functionality.
  • #db_update For any feature that introduces updates to database schema.
  • #deprecation_notice For any upcoming deprecation functionality.
  • #internal For changesets that need to be excluded from the final changelog.
  • #nops For any feature that is NOP facing and needs to be in the official Release Notes for the release.
  • #removed For any functionality/config that is removed.
  • #updated For any functionality that is updated.
  • #wip For any change that is not ready yet and external communication about it should be held off till it is feature complete.

🎖️ No JIRA issue number found in: PR title, commit message, or branch name. Please include the issue ID in one of these.

Copy link
Contributor

github-actions bot commented Dec 10, 2024

AER Report: CI Core ran successfully ✅

aer_workflow , commit

AER Report: Operator UI CI ran successfully ✅

aer_workflow , commit

@@ -398,7 +398,8 @@ func (w *launcher) addToRegistryAndSetDispatcher(ctx context.Context, capability
}

var (
defaultTargetRequestTimeout = time.Minute
// TODO: make this configurable
defaultTargetRequestTimeout = 10 * time.Minute
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does this mean?

if we are writing reports on aptos at up to once every 5 seconds, how can there be a 10 minute timeout in system?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added PR description

@krehermann krehermann added this pull request to the merge queue Dec 10, 2024
Merged via the queue into develop with commit 5f26fd3 Dec 10, 2024
169 of 170 checks passed
@krehermann krehermann deleted the remote_timeout branch December 10, 2024 13:47
bolekk added a commit that referenced this pull request Dec 10, 2024
…anch

Increase default timeout of remote Executable requests (#15587)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants