-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Revert "server: factor out MainCommon as a class, with a run method (… #2613
Conversation
…nvoyproxy#2568)" This reverts commit deffec6, which was causing CI to flake today in TSAN runs. The proximate root cause is the change in envoyproxy#2568, which appears not to correctly setup logging in --mode=validate. This in turn leads to some log related races that I don't fully grok. Rolling back to unblock CI, we should make sure hotrestart_test passes TSAN with high confidence before rolling forward again. Signed-off-by: Harvey Tuch <[email protected]>
@jmarantz @envoyproxy/maintainers Apologies for the rollback, but this will unblock CI. We should try and figure out what's up with |
Sorry about the regression! I wonder if this would've been easier to catch with the thread annotation you've now enabled? |
@jmarantz Yeah, it will be interesting to understand the underlying race; I think somehow we were getting default logging initialization effects rather than a full initialization prior to threads launching (which were then logging). Some of the locking around logging could benefit from thread annotations. |
…method (envoyproxy#2568)" (envoyproxy#2613)" This reverts commit 034135f. Signed-off-by: Joshua Marantz <[email protected]>
…ation (#2623) Description: Reverts #2613 the revert of #2568 Fixed the underlying issue which was that logging was not initialized during validation, so it ran lockless. This caused intermittent tsan errors. Although #2619 makes the failure immediate and consistent by asserting that logging has been initialized prior to spawning any threads. Risk Level: Medium -- the earlier #2568 caused intermittent tsan errors which we believe to be fixed, but #2568 was medium-risk in the first place. Release Notes: N/A
This can save ~1ms at batch size == 100.
Risk Level: low Testing: unit tests Docs Changes: n/a Release Notes: yes Part of #1594 Signed-off-by: Alyssa Wilk <[email protected]> Signed-off-by: JP Simard <[email protected]>
Risk Level: low Testing: unit tests Docs Changes: n/a Release Notes: yes Part of #1594 Signed-off-by: Alyssa Wilk <[email protected]> Signed-off-by: JP Simard <[email protected]>
…#2568)"
This reverts commit deffec6, which was
causing CI to flake today in TSAN runs. The proximate root cause is
the change in #2568, which appears not to correctly setup logging in
--mode=validate. This in turn leads to some log related races that I
don't fully grok.
Rolling back to unblock CI, we should make sure hotrestart_test passes
TSAN with high confidence before rolling forward again.
Risk Level: Low
Testing: bazel test //test/integration:hotrestart_test --runs_per_test=100 --config=clang-tsan -c dbg
Signed-off-by: Harvey Tuch [email protected]