-
Notifications
You must be signed in to change notification settings - Fork 3.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
roachtest: inconsistency failed #54005
Labels
branch-master
Failures and bugs on the master branch.
C-test-failure
Broken test (automatically or manually discovered).
O-roachtest
O-robot
Originated from a bot.
Milestone
Comments
cockroach-teamcity
added
branch-master
Failures and bugs on the master branch.
C-test-failure
Broken test (automatically or manually discovered).
O-roachtest
O-robot
Originated from a bot.
release-blocker
Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.
labels
Sep 7, 2020
tbg
added a commit
to tbg/cockroach
that referenced
this issue
Sep 8, 2020
This test sets up an intentionally corrupted replica and wants its node to shut down as a result of its detection. When only two of the three nodes were included in the consistency check, either one of them could end up terminating (as no obvious majority of healthy replicas can be determined). Change the test so that we wait for the cluster to come fully together before setting a low consistency check interval. Closes cockroachdb#54005. Release justification: testing Release note: None
It's an funny failure. The test basically sets n1 up with an inconsistency and expects it to fatal. But we see n2 fatal, as a result of this consistency check run by n1:
It looks like for some reason n3 isn't included in the check, which can happen if it didn't respond in time or wasn't considered to be live. I think this is easy to avoid, sending PR. |
tbg
removed
the
release-blocker
Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.
label
Sep 8, 2020
craig bot
pushed a commit
that referenced
this issue
Sep 10, 2020
53991: pgwire: accept non-TLS client conns safely in secure mode r=aaron-crl,irfansharif,bdarnell a=knz Fixes #44842. Informs #49532. Informs #53404. This change makes it possible for a DBA / system administrator to reconfigure individual nodes *in a secure cluster* to accept SQL client sessions over TCP without mandating a TLS handshake. Authentication remains mandatory as per the HBA rules. Motivation: we have at least two high-profile customers who keep their nodes and client apps in a private secure network (with network-level encryption / privacy) and who experience client-side TLS as unnecessary and expensive friction. Additionally, **this feature is a prerequisite to upgrade an insecure cluster to secure mode without downtime.** Why this does not impair security: - authentication remains mandatory (as per the HBA rules -- [1] [2]). - the feature is opt-in: the operator must set a command-line flag (`--accept-sql-without-tls`), which is not enabled by default. - there is an interlock: the user must both set up the flag and set log-in passwords for their SQL users (by default, users get created without a password and thus cannot log in without client certs). - for now, node-node connections still require TLS. [1]: https://www.postgresql.org/docs/12/auth-pg-hba-conf.html [2]: https://dr-knz.net/authentication-in-postgresql-and-cockroachdb.html For context, the default HBA configuration is the following: ``` host all root all cert-password # fixed rule host all all all cert-password # built-in CockroachDB default local all all password # built-in CockroachDB default ``` The directive `host` covers both TLS and non-TLS incoming TCP connections (`local` is for the unix socket). The method `cert-password` means "client cert or password": without a cert, the password is mandatory. As previously, the user can further secure the configuration by restricting non-TLS connections to just a subnetwork, for example: ``` host all all 10.0.0.0/8 password # accept conns on the 10/8 network. host all all all reject # refuse conns from other nets. local all all password ``` Note that this change is limited to the server side: CockroachDB's own `cockroach` CLI commands do not yet know how to connect to a CockroachDB server without TLS; such connections are only supported from `psql` or SQL client drivers in apps. See #53994 for a follow-up. Release justification: fixes for high-priority or high-severity bugs in existing functionality 54019: roachtest: de-flake 'inconsistent' r=knz a=tbg This test sets up an intentionally corrupted replica and wants its node to shut down as a result of its detection. When only two of the three nodes were included in the consistency check, either one of them could end up terminating (as no obvious majority of healthy replicas can be determined). Change the test so that we wait for the cluster to come fully together before setting a low consistency check interval. Closes #54005. Release justification: testing Release note: None Co-authored-by: Raphael 'kena' Poss <knz@thaumogen.net> Co-authored-by: Tobias Grieger <tobias.b.grieger@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
branch-master
Failures and bugs on the master branch.
C-test-failure
Broken test (automatically or manually discovered).
O-roachtest
O-robot
Originated from a bot.
(roachtest).inconsistency failed on master@8b9e8dc32e73cfdfc1999da35d61e5cc9a2b35ec:
More
Artifacts: /inconsistency
See this test on roachdash
powered by pkg/cmd/internal/issues
The text was updated successfully, but these errors were encountered: