Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sql/pgwire: TestAuthenticationAndHBARules failed #131110

Closed
cockroach-teamcity opened this issue Sep 20, 2024 · 2 comments · Fixed by #135086, mohini-crl/cockroach#34, mohini-crl/cockroach#128 or mohini-crl/cockroach#138
Closed

sql/pgwire: TestAuthenticationAndHBARules failed #131110

cockroach-teamcity opened this issue Sep 20, 2024 · 2 comments · Fixed by #135086, mohini-crl/cockroach#34, mohini-crl/cockroach#128 or mohini-crl/cockroach#138
Assignees
@souravcrl
Labels
branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 branch-release-24.3 Used to mark GA and release blockers, technical advisories, and bugs for 24.3 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-product-security
Milestone
23.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Sep 20, 2024 •
edited by cockroach-jira-scripts
Loading

sql/pgwire.TestAuthenticationAndHBARules failed with artifacts on release-23.2 @ 9fc163504ad3f06dc1dd0838493c8e1b6ffc4f72:

        config [1 args]
        <no input to command>
        ----
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1318/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:12:
        set_hba [0 args]
        host  all root all cert-password
        host  all all  all cert-password
        local all all      password
        ----
        # Active authentication configuration on this node:
        # Original configuration:
        # loopback all all all trust       # built-in CockroachDB default
        # host  all root all cert-password
        # host  all all  all cert-password
        # local all all      password
        #
        # Interpreted configuration:
        # TYPE   DATABASE USER ADDRESS METHOD        OPTIONS
        loopback all      all  all     trust
        host     all      root all     cert-password
        host     all      all  all     cert-password
        local    all      all          password
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/28616e575ae934b92eba220441c7ca42/logTestAuthenticationAndHBARules_insecure=false_hba_default_equivalence1610180880
        --- FAIL: TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence (1.23s)
=== RUN   TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence/root
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1318/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:34:
        connect [1 args]
        <no input to command>
        ----
        ok defaultdb
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1318/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:40:
        connect_unix [1 args]
        <no input to command>
        ----
        ERROR: password authentication failed for user root (SQLSTATE 28P01)
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1318/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:46:
         
        expected:
        ERROR: password authentication failed for user root (SQLSTATE 28P01)
        
        found:
        ERROR: pq: SSL is not enabled on the server
            --- FAIL: TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence/root (0.03s)
=== RUN   TestAuthenticationAndHBARules/insecure=false
    --- FAIL: TestAuthenticationAndHBARules/insecure=false (24.10s)

Parameters:

  • TAGS=bazel,gss
  • stress=true
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

/cc @cockroachdb/sql-foundations @cockroachdb/server

This test on roachdash | Improve this report!

Jira issue: CRDB-42378
@cockroach-teamcity cockroach-teamcity added branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) labels Sep 20, 2024
@cockroach-teamcity cockroach-teamcity added this to the 23.2 milestone Sep 20, 2024
@fqazi fqazi added P-3 Issues/test failures with no fix SLA P-2 Issues/test failures with a fix SLA of 3 months and removed P-3 Issues/test failures with no fix SLA release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Sep 24, 2024
@cockroach-teamcity cockroach-teamcity mentioned this issue Sep 27, 2024
Closed
souravcrl added a commit to souravcrl/cockroach that referenced this issue Sep 30, 2024
@souravcrl
sql: fix TestAuthenticationAndHBARules for special_cases data-driven …
7713d36 
…test

informs cockroachdb#131532
informs cockroachdb#131110
informs cockroachdb#130253
informs cockroachdb#127745
Epic: CRDB-41958

`TestAuthenticationAndHBARules` fails for special_cases data driven test. We
suspect it might be due to client for `special_cases` test accessing the test
server from a previous test `secure_non_tls` which sets `accept_sql_without_tls`
to true. This results in the following error `ERROR: pq: SSL is not enabled on
the server` while the client was expecting an SSL connection with the server. We
fix this in the PR.

Release note: None
@souravcrl souravcrl mentioned this issue Sep 30, 2024
Merged
@exalate-issue-sync exalate-issue-sync bot added T-product-security and removed T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) labels Sep 30, 2024
craig bot pushed a commit that referenced this issue Oct 1, 2024
@souravcrl
Merge #131580
90e4ba0 
131580: sql: fix TestAuthenticationAndHBARules for special_cases data-driven test r=rafiss a=souravcrl

informs #131532
informs #131110
informs #130253
informs #127745
Epic: CRDB-41958

`TestAuthenticationAndHBARules` fails for special_cases data driven test. We suspect it might be due to client for `special_cases` test accessing the test server from a previous test `secure_non_tls` which sets `accept_sql_without_tls` to true. This results in the following error `ERROR: pq: SSL is not enabled on the server` while the client was expecting an SSL connection with the server. We fix this in the PR.

Release note: None

Co-authored-by: souravcrl <[email protected]>
@blathers-crl blathers-crl bot mentioned this issue Oct 1, 2024
Merged
blathers-crl bot pushed a commit that referenced this issue Oct 1, 2024
@souravcrl
sql: fix TestAuthenticationAndHBARules for special_cases data-driven …
75cb1a9 
…test

informs #131532
informs #131110
informs #130253
informs #127745
Epic: CRDB-41958

`TestAuthenticationAndHBARules` fails for special_cases data driven test. We
suspect it might be due to client for `special_cases` test accessing the test
server from a previous test `secure_non_tls` which sets `accept_sql_without_tls`
to true. This results in the following error `ERROR: pq: SSL is not enabled on
the server` while the client was expecting an SSL connection with the server. We
fix this in the PR.

Release note: None
@blathers-crl blathers-crl bot mentioned this issue Oct 1, 2024
Merged
blathers-crl bot pushed a commit that referenced this issue Oct 1, 2024
@souravcrl
sql: fix TestAuthenticationAndHBARules for special_cases data-driven …
3683b57 
…test

informs #131532
informs #131110
informs #130253
informs #127745
Epic: CRDB-41958

`TestAuthenticationAndHBARules` fails for special_cases data driven test. We
suspect it might be due to client for `special_cases` test accessing the test
server from a previous test `secure_non_tls` which sets `accept_sql_without_tls`
to true. This results in the following error `ERROR: pq: SSL is not enabled on
the server` while the client was expecting an SSL connection with the server. We
fix this in the PR.

Release note: None
@blathers-crl blathers-crl bot mentioned this issue Oct 1, 2024
Merged
blathers-crl bot pushed a commit that referenced this issue Oct 1, 2024
@souravcrl
sql: fix TestAuthenticationAndHBARules for special_cases data-driven …
302524c 
…test

informs #131532
informs #131110
informs #130253
informs #127745
Epic: CRDB-41958

`TestAuthenticationAndHBARules` fails for special_cases data driven test. We
suspect it might be due to client for `special_cases` test accessing the test
server from a previous test `secure_non_tls` which sets `accept_sql_without_tls`
to true. This results in the following error `ERROR: pq: SSL is not enabled on
the server` while the client was expecting an SSL connection with the server. We
fix this in the PR.

Release note: None
@blathers-crl blathers-crl bot mentioned this issue Oct 1, 2024
Merged
souravcrl added a commit that referenced this issue Oct 1, 2024
@souravcrl
sql: fix TestAuthenticationAndHBARules for special_cases data-driven …
60da1fe 
…test

informs #131532
informs #131110
informs #130253
informs #127745
Epic: CRDB-41958

`TestAuthenticationAndHBARules` fails for special_cases data driven test. We
suspect it might be due to client for `special_cases` test accessing the test
server from a previous test `secure_non_tls` which sets `accept_sql_without_tls`
to true. This results in the following error `ERROR: pq: SSL is not enabled on
the server` while the client was expecting an SSL connection with the server. We
fix this in the PR.

Release note: None
This was referenced Oct 2, 2024
Closed
Closed
souravcrl added a commit to souravcrl/cockroach that referenced this issue Oct 16, 2024
@souravcrl
informs cockroachdb#131532
69b511c 
informs cockroachdb#131110
informs cockroachdb#130253
informs cockroachdb#127745
Epic CRDB-41958

TestAuthenticationAndHBARules fails for `special_cases`,
`hba_default_equivalence`, `empty_hba` data driven tests for secure mode. The
failures occur when root user is trying to authenticate with cert-password auth
method and `sslmode` is set to `verify-ca` with `sslcert` being empty. The
expected behavior is root authentication defaults to password method and fails
as no password is set for root, but instead we get:
```
SSL is not enabled on the server
```
Since the failures are there only under stress, it might be because db server
shutdown or paused before responding to request for upgrade connection to SSL
from lib/pq client from here
https://github.com/lib/pq/blob/3d613208bca2e74f2a20e04126ed30bcb5c4cc27/conn.go#L1116-L1130.
Retrying connection establishment when this specific error is obtained might fix
the problem as this logic seems faulty(it checks for absence of 'S' in server
response whereas the correct check should be for 'N' in response).

Release note: None
@cockroach-teamcity cockroach-teamcity mentioned this issue Oct 16, 2024
Closed
souravcrl added a commit to souravcrl/cockroach that referenced this issue Oct 18, 2024
@souravcrl
informs cockroachdb#131532
e3ede16 
informs cockroachdb#131110
informs cockroachdb#130253
informs cockroachdb#127745
Epic CRDB-41958

TestAuthenticationAndHBARules fails for `special_cases`,
`hba_default_equivalence`, `empty_hba` data driven tests for secure mode. The
failures occur when root user is trying to authenticate with cert-password auth
method and `sslmode` is set to `verify-ca` with `sslcert` being empty. The
expected behavior is root authentication defaults to password method and fails
as no password is set for root, but instead we get:
```
SSL is not enabled on the server
```
Since the failures are there only under stress, it might be because db server
shutdown or paused before responding to request for upgrade connection to SSL
from lib/pq client from here
https://github.com/lib/pq/blob/3d613208bca2e74f2a20e04126ed30bcb5c4cc27/conn.go#L1116-L1130.
Retrying connection establishment when this specific error is obtained might fix
the problem as this logic seems faulty(it checks for absence of 'S' in server
response whereas the correct check should be for 'N' in response).

Release note: None
souravcrl added a commit to souravcrl/cockroach that referenced this issue Oct 19, 2024
@souravcrl
sql: fix TestAuthenticationAndHBARules stress tests SSL issue
5e62c2e 
informs cockroachdb#131532
informs cockroachdb#131110
informs cockroachdb#130253
informs cockroachdb#127745
Epic CRDB-41958

TestAuthenticationAndHBARules fails for `special_cases`,
`hba_default_equivalence`, `empty_hba` data driven tests for secure mode. The
failures occur when root user is trying to authenticate with cert-password auth
method and `sslmode` is set to `verify-ca` with `sslcert` being empty. The
expected behavior is root authentication defaults to password method and fails
as no password is set for root, but instead we get:
```
SSL is not enabled on the server
```
Since the failures are there only under stress, it might be because db server
shutdown or paused before responding to request for upgrade connection to SSL
from lib/pq client from here
https://github.com/lib/pq/blob/3d613208bca2e74f2a20e04126ed30bcb5c4cc27/conn.go#L1116-L1130.
Retrying connection establishment when this specific error is obtained might fix
the problem as this logic seems faulty(it checks for absence of 'S' in server
response whereas the correct check should be for 'N' in response).

Release note: None
@souravcrl souravcrl mentioned this issue Oct 19, 2024
Closed
@cockroach-teamcity
Copy link
Member Author

sql/pgwire.TestAuthenticationAndHBARules failed with artifacts on release-23.2 @ 77237763413fff744e598f85a8b554ac3de66c38:

        HINT: consider using .ApplicationLayer().GetAdminHTTPClient() instead.
    conditional_wrap.go:189: TIP: consider replacing the test server initialization from:
            ts, ... := serverutils.StartServer(t, ...)
            defer ts.Stopper().Stop(...)
        to:
            srv, ... := serverutils.StartServer(t, ...)
            defer srv.Stopper().Stop(...)
            ts := srv.ApplicationLayer()
        
        See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/19/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/special_cases:3:
        config [1 args]
        <no input to command>
        ----
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/19/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/special_cases:6:
        sql [0 args]
        CREATE USER passworduser WITH PASSWORD 'pass'
        ----
        ok
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/28616e575ae934b92eba220441c7ca42/logTestAuthenticationAndHBARules_insecure=false_special_cases2614998304
        --- FAIL: TestAuthenticationAndHBARules/insecure=false/special_cases (1.39s)
=== RUN   TestAuthenticationAndHBARules/insecure=false/special_cases/root_user_cannot_use_password
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/19/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/special_cases:17:
        set_hba [0 args]
        host all root 0.0.0.0/0 password
        ----
        # Active authentication configuration on this node:
        # Original configuration:
        # loopback all all all trust       # built-in CockroachDB default
        # host  all root all cert-password # CockroachDB mandatory rule
        # host all root 0.0.0.0/0 password
        #
        # Interpreted configuration:
        # TYPE   DATABASE USER ADDRESS   METHOD        OPTIONS
        loopback all      all  all       trust
        host     all      root all       cert-password
        host     all      root 0.0.0.0/0 password
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/19/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/special_cases:32:
         
        expected:
        ERROR: password authentication failed for user root (SQLSTATE 28P01)
        
        found:
        ERROR: pq: SSL is not enabled on the server
            --- FAIL: TestAuthenticationAndHBARules/insecure=false/special_cases/root_user_cannot_use_password (0.02s)

Parameters:

  • TAGS=bazel,gss
  • stress=true
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Oct 24, 2024
Closed
@rafiss rafiss mentioned this issue Oct 29, 2024
Merged
craig bot pushed a commit that referenced this issue Oct 29, 2024
@rafiss
Merge #133688
f759a46 
133688: pgwire: add more test logs to debug flaky test r=rafiss a=rafiss

informs #127745
informs #133360
informs #131532
informs #131110
Release note: None

Co-authored-by: Rafi Shamim <[email protected]>
This was referenced Oct 29, 2024
Merged
Merged
Merged
Merged
Merged
@rafiss rafiss mentioned this issue Nov 13, 2024
Merged
@craig craig bot closed this as completed in afd5d06 Nov 14, 2024
@blathers-crl blathers[crl]
Copy link

blathers-crl bot commented Nov 14, 2024

Based on the specified backports for linked PR #135086, I applied the following new label(s) to this issue: branch-release-23.1, branch-release-24.1, branch-release-24.2, branch-release-24.3. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@blathers-crl blathers-crl bot added branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 branch-release-24.3 Used to mark GA and release blockers, technical advisories, and bugs for 24.3 labels Nov 14, 2024
@blathers-crl blathers-crl bot mentioned this issue Nov 14, 2024
Merged
This was referenced Nov 14, 2024
Merged
Merged
Merged
Merged
This was referenced Dec 19, 2024
Merged
Merged
This was referenced Jan 5, 2025
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@souravcrl souravcrl

Labels
branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 branch-release-24.3 Used to mark GA and release blockers, technical advisories, and bugs for 24.3 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-product-security
Projects
None yet
Milestone
23.2
3 participants
@fqazi @cockroach-teamcity @souravcrl