Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: envoy shutdown flaky test #3646

Merged
merged 1 commit into from
Jun 22, 2024
Merged

Conversation

guydc
Copy link
Contributor

@guydc guydc commented Jun 21, 2024

What this PR does / why we need it:

This PR delays readiness of envoy to allow XDS sync as discussed in #2810. This is done specifically for the upgrade test suite gateway class.

Test flakiness seems to be related to 404 responses in early stages of the load test, which likely indicates that new proxy instances are ready before being programmed.

--- FAIL: TestEGUpgrade (230.50s)
    --- PASS: TestEGUpgrade/EGUpgrade (152.90s)
        --- PASS: TestEGUpgrade/EGUpgrade/Upgrade_from_an_older_eg_release_should_succeed (152.89s)
    --- FAIL: TestEGUpgrade/EnvoyShutdown (68.06s)
        --- FAIL: TestEGUpgrade/EnvoyShutdown/All_requests_must_succeed (68.04s)
        utils.go:176: failed to create load: error 404 for http://172.18.255.208/envoy-shutdown (82 bytes)
        utils.go:185: Load completed after 0s with 2 requests, 0 success, 2 failures and 2 errors

Which issue(s) this PR fixes:
Fixes #3262

@guydc guydc requested a review from a team as a code owner June 21, 2024 18:33
Copy link
Contributor

@arkodg arkodg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM thanks !

@arkodg arkodg requested review from a team June 21, 2024 18:34
@arkodg
Copy link
Contributor

arkodg commented Jun 21, 2024

the 5s delay may be helpful for this e2e, but will need to be tweaked for user environments where the xDS config is large and many envoys are connecting to the control plane at once
A more robust solution would be to use the listener level health check recently added https://gateway.envoyproxy.io/latest/api/extension_types/#healthchecksettings as a active health check in the external loadbalancer. I dont think this can be added to metal lb, but will be useful for cloud environments

Copy link

codecov bot commented Jun 21, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 68.30%. Comparing base (c752b88) to head (5ea7a67).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3646      +/-   ##
==========================================
+ Coverage   68.27%   68.30%   +0.03%     
==========================================
  Files         170      170              
  Lines       20760    20760              
==========================================
+ Hits        14174    14181       +7     
+ Misses       5566     5560       -6     
+ Partials     1020     1019       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@arkodg arkodg merged commit 432b7e3 into envoyproxy:main Jun 22, 2024
25 checks passed
bjlhlin pushed a commit to bjlhlin/gateway that referenced this pull request Jun 24, 2024
delay readdiness to allow XDS sync

Signed-off-by: Guy Daich <[email protected]>
Signed-off-by: “bjlhlin” <“[email protected]”>
bjlhlin pushed a commit to bjlhlin/gateway that referenced this pull request Jun 24, 2024
delay readdiness to allow XDS sync

Signed-off-by: Guy Daich <[email protected]>
Signed-off-by: “bjlhlin” <“[email protected]”>
Signed-off-by: bjlhlin <[email protected]>
bjlhlin pushed a commit to bjlhlin/gateway that referenced this pull request Jun 24, 2024
delay readdiness to allow XDS sync

Signed-off-by: Guy Daich <[email protected]>
Signed-off-by: “bjlhlin” <“[email protected]”>
Signed-off-by: bjlhlin <[email protected]>
bjlhlin pushed a commit to bjlhlin/gateway that referenced this pull request Jun 24, 2024
delay readdiness to allow XDS sync

Signed-off-by: Guy Daich <[email protected]>
Signed-off-by: “bjlhlin” <“[email protected]”>
Signed-off-by: bjlhlin <[email protected]>
bjlhlin pushed a commit to bjlhlin/gateway that referenced this pull request Jun 24, 2024
delay readdiness to allow XDS sync

Signed-off-by: Guy Daich <[email protected]>
Signed-off-by: “bjlhlin” <“[email protected]”>
Signed-off-by: bjlhlin <[email protected]>
bjlhlin pushed a commit to bjlhlin/gateway that referenced this pull request Jun 24, 2024
delay readdiness to allow XDS sync

Signed-off-by: Guy Daich <[email protected]>
Signed-off-by: “bjlhlin” <“[email protected]”>
Signed-off-by: bjlhlin <[email protected]>
bjlhlin pushed a commit to bjlhlin/gateway that referenced this pull request Jun 25, 2024
delay readdiness to allow XDS sync

Signed-off-by: Guy Daich <[email protected]>
Signed-off-by: bjlhlin <[email protected]>
bjlhlin pushed a commit to bjlhlin/gateway that referenced this pull request Jun 26, 2024
delay readdiness to allow XDS sync

Signed-off-by: Guy Daich <[email protected]>
Signed-off-by: bjlhlin <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

flaky: e2e TestEGUpgrade/EnvoyShutdown
3 participants