Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase how long we wait for the api pod to come up #157

Merged

Conversation

rooftopcellist
Copy link
Member

Resolves: #154

This PR increases how long we wait for the api pod to come up

  • There are 3 initContainers and migrations that need to run before the API pod is in the running state, this is expected to take a bit of time.
  • This will avoid the reconciliation loop failing when checking the API pod (default was only 15 seconds prior to this change).

cc @kurokobo

@kurokobo
Copy link
Contributor

Thanks for working on this!
I have confirmed that it works as expected; the first loop completes without failure.

Just to confirm, is it the expected behavior that the task succeeds and moves on even if the pod is 1/2 Running?

  • eda_api_pod['resources'][0]['status']['containerStatuses'][0] appears to be daphne, not eda-api (containerStatuses are ordered by name).
  • In the current implementation, if daphne instead of eda-api becomes ready: true, the task succeeds.
  • The startup of daphne completes immediately, but eda-api will not be ready: true for at least 90 seconds because of the readinessProbe with initialDelaySeconds: 90.

Should we wait until it is 2/2 Running before proceeding like this? Also initialDelaySeconds could be a little shorter.

  until:
    - "eda_api_pod['resources'] | length"
    - "eda_api_pod['resources'][0]['status']['phase'] == 'Running'"
    - "eda_api_pod['resources'][0]['status']['containerStatuses'][0]['ready'] == true"
    - "eda_api_pod['resources'][0]['status']['containerStatuses'][1]['ready'] == true"     👈👈👈

Again, even if we proceed at 1/2 Running (only daphne is ready), the deployment will complete without any issues, since probably the startup of eda-api is completed internally within 90 seconds.

- There are 3 initContainers and migrations that need to run before the
  API pod is in the running state, this is expected to take a bit of
  time.
- This will avoid the reconciliation loop failing when checking the API
  pod (default was only 15 seconds prior to this change).
- Reduce the initialDelaySeconds from 90 to 20 seconds for the api pod

Signed-off-by: Christian M. Adams <[email protected]>
@rooftopcellist rooftopcellist force-pushed the increase-wait-for-api-pod branch from 5675f42 to b89dbe9 Compare January 6, 2024 19:50
@rooftopcellist
Copy link
Member Author

rooftopcellist commented Jan 6, 2024

Thank you for the feedback and suggestions @kurokobo! I too see that the task moves on before both containers in the api pod are in running state. I think we should wait for both to be in the running state.

I reduced the initialDelaySeconds to 20 seconds and added the bit to wait for both containers of the api pod. Thanks!

@rooftopcellist rooftopcellist merged commit 178da90 into ansible:main Jan 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

On fresh deployments, sometimes the first reconciliation loop fails on the task "Check for API Pod"
3 participants