Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature] Resource Manager should trigger Exponential backoff for lack of resources #442

Closed
2 of 13 tasks
EngHabu opened this issue Aug 4, 2020 · 3 comments
Closed
2 of 13 tasks

[Feature] Resource Manager should trigger Exponential backoff for lack of resources #442

EngHabu opened this issue Aug 4, 2020 · 3 comments
Labels
enhancement New feature or request stale untriaged This issues has not yet been looked at by the Maintainers

Comments

@EngHabu
Copy link
Contributor

EngHabu commented Aug 4, 2020

Motivation: Why do you think this is important?
When Resource Manager.AllocateToken() fails (due to lack of resources), propeller will keep traversing the same workflow over and over again attempting to allocate token and wasting time/resources.

Goal: What should the final outcome look like, ideally?
When AllocateToken fails, it should trigger exponential backoff and hold off attempting to allocate tokens to relieve pressure on the workers.

Flyte component

  • Overall
  • Flyte Setup and Installation scripts
  • Flyte Documentation
  • Flyte communication (slack/email etc)
  • FlytePropeller
  • FlyteIDL (Flyte specification language)
  • Flytekit (Python SDK)
  • FlyteAdmin (Control Plane service)
  • FlytePlugins
  • DataCatalog
  • FlyteStdlib (common libraries)
  • FlyteConsole (UI)
  • Other
@EngHabu EngHabu added enhancement New feature or request untriaged This issues has not yet been looked at by the Maintainers labels Aug 4, 2020
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 6, 2022
@EngHabu
Strip Type Metadata (flyteorg#442)
755e90a 
* Strip Type Metadata

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Strip variable descriptions

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Strip Description

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Use a fixed time for datetime field literals

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Move logic to strip metadata to k8s transformer

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Update code documentation

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* PR Comments

Signed-off-by: Haytham Abuelfutuh <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 6, 2022
@katrogan
Add notification validation (flyteorg#442)
7b48c85 
* checkpoint

Signed-off-by: Katrina Rogan <[email protected]>

* lint

Signed-off-by: Katrina Rogan <[email protected]>

* rename

Signed-off-by: Katrina Rogan <[email protected]>

* more tests, fine

Signed-off-by: Katrina Rogan <[email protected]>

* cleanup

Signed-off-by: Katrina Rogan <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 20, 2022
@wild-endeavor
[nit] Ignore bash history (flyteorg#442)
aba2925 
Signed-off-by: Yee Hing Tong <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Aug 9, 2023
@EngHabu
Strip Type Metadata (flyteorg#442)
bc060ea 
* Strip Type Metadata

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Strip variable descriptions

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Strip Description

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Use a fixed time for datetime field literals

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Move logic to strip metadata to k8s transformer

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Update code documentation

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* PR Comments

Signed-off-by: Haytham Abuelfutuh <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Aug 21, 2023
@katrogan
Add notification validation (flyteorg#442)
047a80f 
* checkpoint

Signed-off-by: Katrina Rogan <[email protected]>

* lint

Signed-off-by: Katrina Rogan <[email protected]>

* rename

Signed-off-by: Katrina Rogan <[email protected]>

* more tests, fine

Signed-off-by: Katrina Rogan <[email protected]>

* cleanup

Signed-off-by: Katrina Rogan <[email protected]>
@github-actions
Copy link

Hello 👋, This issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will close the issue if we detect no activity in the next 7 days. Thank you for your contribution and understanding! 🙏

@github-actions github-actions bot added the stale label Aug 26, 2023
@github-actions
Copy link

github-actions bot commented Sep 2, 2023

Hello 👋, This issue has been inactive for over 9 months and hasn't received any updates since it was marked as stale. We'll be closing this issue for now, but if you believe this issue is still relevant, please feel free to reopen it. Thank you for your contribution and understanding! 🙏

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Sep 2, 2023
@eapolinario eapolinario reopened this Nov 2, 2023
@kumare3
Copy link
Contributor

kumare3 commented Dec 22, 2023

This is already supported.

@kumare3 kumare3 closed this as completed Dec 22, 2023
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Apr 30, 2024
@jeevb
Increase sleep seconds when waiting for sandbox (flyteorg#442)
d297945 
Signed-off-by: Jeev B <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Apr 30, 2024
@jeevb
Increase sleep seconds when waiting for sandbox (flyteorg#442)
9553923 
Signed-off-by: Jeev B <[email protected]>
austin362667 pushed a commit to austin362667/flyte that referenced this issue May 7, 2024
@jeevb @austin362667
Increase sleep seconds when waiting for sandbox (flyteorg#442)
c9d734c 
Signed-off-by: Jeev B <[email protected]>
robert-ulbrich-mercedes-benz pushed a commit to robert-ulbrich-mercedes-benz/flyte that referenced this issue Jul 2, 2024
@jeevb @robert-ulbrich-mercedes-benz
Increase sleep seconds when waiting for sandbox (flyteorg#442)
618f1bd 
Signed-off-by: Jeev B <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request stale untriaged This issues has not yet been looked at by the Maintainers
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@EngHabu @eapolinario @kumare3