sql/pgwire: TestAuthenticationAndHBARules failed #127745

cockroach-teamcity · 2024-07-26T17:13:47Z

sql/pgwire.TestAuthenticationAndHBARules failed with artifacts on release-23.1 @ 3314a3e81d78361ba4ac45fc52a84efdfb3466ed:

        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2917/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:9:
        config [1 args]
        <no input to command>
        ----
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2917/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:12:
        set_hba [0 args]
        host  all root all cert-password
        host  all all  all cert-password
        local all all      password
        ----
        # Active authentication configuration on this node:
        # Original configuration:
        # loopback all all all trust       # built-in CockroachDB default
        # host  all root all cert-password
        # host  all all  all cert-password
        # local all all      password
        #
        # Interpreted configuration:
        # TYPE   DATABASE USER ADDRESS METHOD        OPTIONS
        loopback all      all  all     trust
        host     all      root all     cert-password
        host     all      all  all     cert-password
        local    all      all          password
=== CONT  TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/28616e575ae934b92eba220441c7ca42/logTestAuthenticationAndHBARules_insecure=false_hba_default_equivalence2560213530
        --- FAIL: TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence (1.11s)
=== RUN   TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence/root
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2917/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:34:
        connect [1 args]
        <no input to command>
        ----
        ok defaultdb
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2917/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:40:
        connect_unix [1 args]
        <no input to command>
        ----
        ERROR: password authentication failed for user root (SQLSTATE 28P01)
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2917/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:46:
         
        expected:
        ERROR: password authentication failed for user root (SQLSTATE 28P01)
        
        found:
        ERROR: pq: SSL is not enabled on the server
            --- FAIL: TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence/root (0.02s)

Parameters:

TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

sql/pgwire: TestAuthenticationAndHBARules failed #126758 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-23.1.24-rc]
sql/pgwire: TestAuthenticationAndHBARules failed #126239 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-23.2]
sql/pgwire: TestAuthenticationAndHBARules failed #126063 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-23.2.7-rc]

/cc @cockroachdb/sql-foundations @cockroachdb/server _{This test on roachdash | Improve this report!

Jira issue: CRDB-40580}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2024-08-14T20:31:39Z

sql/pgwire.TestAuthenticationAndHBARules failed with artifacts on release-23.1 @ df2118610b0ea8cc174f19717129882e24c517b3:

        <no input to command>
        ----
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2917/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:12:
        set_hba [0 args]
        host  all root all cert-password
        host  all all  all cert-password
        local all all      password
        ----
        # Active authentication configuration on this node:
        # Original configuration:
        # loopback all all all trust       # built-in CockroachDB default
        # host  all root all cert-password
        # host  all all  all cert-password
        # local all all      password
        #
        # Interpreted configuration:
        # TYPE   DATABASE USER ADDRESS METHOD        OPTIONS
        loopback all      all  all     trust
        host     all      root all     cert-password
        host     all      all  all     cert-password
        local    all      all          password
=== CONT  TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/28616e575ae934b92eba220441c7ca42/logTestAuthenticationAndHBARules_insecure=false_hba_default_equivalence2438763659
        --- FAIL: TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence (1.21s)
=== RUN   TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence/root
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2917/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:34:
        connect [1 args]
        <no input to command>
        ----
        ok defaultdb
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2917/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:40:
        connect_unix [1 args]
        <no input to command>
        ----
        ERROR: password authentication failed for user root (SQLSTATE 28P01)
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2917/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:46:
         
        expected:
        ERROR: password authentication failed for user root (SQLSTATE 28P01)
        
        found:
        ERROR: pq: SSL is not enabled on the server
            --- FAIL: TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence/root (0.01s)
=== RUN   TestAuthenticationAndHBARules/insecure=false
    --- FAIL: TestAuthenticationAndHBARules/insecure=false (23.91s)

Parameters:

TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

sql/pgwire: TestAuthenticationAndHBARules failed #128765 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-23.2.9-rc]
sql/pgwire: TestAuthenticationAndHBARules failed #128094 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-master]
sql/pgwire: TestAuthenticationAndHBARules failed #126758 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-23.1.24-rc no-test-failure-activity]
sql/pgwire: TestAuthenticationAndHBARules failed #126239 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-23.2]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2024-08-28T20:50:15Z

sql/pgwire.TestAuthenticationAndHBARules failed with artifacts on release-23.1 @ 52fbd4aee17b0fe61dfb3badd2401fd3b1430bf2:

        connect [1 args]
        <no input to command>
        ----
        ok defaultdb
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/812/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/empty_hba:45:
        connect_unix [1 args]
        <no input to command>
        ----
        ERROR: password authentication failed for user root (SQLSTATE 28P01)
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/812/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/empty_hba:51:
         
        expected:
        ERROR: password authentication failed for user root (SQLSTATE 28P01)
        
        found:
        ERROR: pq: SSL is not enabled on the server
            --- FAIL: TestAuthenticationAndHBARules/insecure=false/empty_hba/root (0.02s)
=== RUN   TestAuthenticationAndHBARules/insecure=false
    --- FAIL: TestAuthenticationAndHBARules/insecure=false (22.63s)
=== RUN   TestAuthenticationAndHBARules/insecure=false/empty_hba
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/28616e575ae934b92eba220441c7ca42/logTestAuthenticationAndHBARules_insecure=false_empty_hba258861039
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/812/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/empty_hba:16:
        config [1 args]
        <no input to command>
        ----
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/812/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/empty_hba:20:
        set_hba [0 args]
        <no input to command>
        ----
        # Active authentication configuration on this node:
        # Original configuration:
        # host  all root all cert-password # CockroachDB mandatory rule
        # loopback all all all trust       # built-in CockroachDB default
        # host     all all all cert-password # built-in CockroachDB default
        # local    all all     password      # built-in CockroachDB default
        #
        # Interpreted configuration:
        # TYPE   DATABASE USER ADDRESS METHOD        OPTIONS
        host     all      root all     cert-password
        loopback all      all  all     trust
        host     all      all  all     cert-password
        local    all      all          password
=== CONT  TestAuthenticationAndHBARules/insecure=false/empty_hba
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/28616e575ae934b92eba220441c7ca42/logTestAuthenticationAndHBARules_insecure=false_empty_hba258861039
        --- FAIL: TestAuthenticationAndHBARules/insecure=false/empty_hba (1.24s)

Parameters:

TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

sql/pgwire: TestAuthenticationAndHBARules failed #129181 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-23.2.10-rc]
sql/pgwire: TestAuthenticationAndHBARules failed #128765 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-23.2.9-rc]
sql/pgwire: TestAuthenticationAndHBARules failed #128094 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-master]
sql/pgwire: TestAuthenticationAndHBARules failed #126239 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-23.2 no-test-failure-activity]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2024-09-10T16:37:14Z

sql/pgwire.TestAuthenticationAndHBARules failed with artifacts on release-23.1 @ 2381c25c1d4b743b692b772c6777805bb06920ea:

    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/28616e575ae934b92eba220441c7ca42/logTestAuthenticationAndHBARules_insecure=false_empty_hba1152177290
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2927/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/empty_hba:16:
        config [1 args]
        <no input to command>
        ----
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2927/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/empty_hba:20:
        set_hba [0 args]
        <no input to command>
        ----
        # Active authentication configuration on this node:
        # Original configuration:
        # host  all root all cert-password # CockroachDB mandatory rule
        # loopback all all all trust       # built-in CockroachDB default
        # host     all all all cert-password # built-in CockroachDB default
        # local    all all     password      # built-in CockroachDB default
        #
        # Interpreted configuration:
        # TYPE   DATABASE USER ADDRESS METHOD        OPTIONS
        host     all      root all     cert-password
        loopback all      all  all     trust
        host     all      all  all     cert-password
        local    all      all          password
=== CONT  TestAuthenticationAndHBARules/insecure=false/empty_hba
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/28616e575ae934b92eba220441c7ca42/logTestAuthenticationAndHBARules_insecure=false_empty_hba1152177290
        --- FAIL: TestAuthenticationAndHBARules/insecure=false/empty_hba (1.29s)
=== RUN   TestAuthenticationAndHBARules/insecure=false/empty_hba/root
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2927/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/empty_hba:39:
        connect [1 args]
        <no input to command>
        ----
        ok defaultdb
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2927/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/empty_hba:45:
        connect_unix [1 args]
        <no input to command>
        ----
        ERROR: password authentication failed for user root (SQLSTATE 28P01)
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2927/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/empty_hba:51:
         
        expected:
        ERROR: password authentication failed for user root (SQLSTATE 28P01)
        
        found:
        ERROR: pq: SSL is not enabled on the server
            --- FAIL: TestAuthenticationAndHBARules/insecure=false/empty_hba/root (0.02s)

Parameters:

TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

sql/pgwire: TestAuthenticationAndHBARules failed #130253 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot T-sql-foundations branch-release-23.1.26-rc release-blocker]
sql/pgwire: TestAuthenticationAndHBARules failed #129181 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-23.2.10-rc]
sql/pgwire: TestAuthenticationAndHBARules failed #128765 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-23.2.9-rc]

_{This test on roachdash | Improve this report!}

…test informs cockroachdb#131532 informs cockroachdb#131110 informs cockroachdb#130253 informs cockroachdb#127745 Epic: CRDB-41958 `TestAuthenticationAndHBARules` fails for special_cases data driven test. We suspect it might be due to client for `special_cases` test accessing the test server from a previous test `secure_non_tls` which sets `accept_sql_without_tls` to true. This results in the following error `ERROR: pq: SSL is not enabled on the server` while the client was expecting an SSL connection with the server. We fix this in the PR. Release note: None

131580: sql: fix TestAuthenticationAndHBARules for special_cases data-driven test r=rafiss a=souravcrl informs #131532 informs #131110 informs #130253 informs #127745 Epic: CRDB-41958 `TestAuthenticationAndHBARules` fails for special_cases data driven test. We suspect it might be due to client for `special_cases` test accessing the test server from a previous test `secure_non_tls` which sets `accept_sql_without_tls` to true. This results in the following error `ERROR: pq: SSL is not enabled on the server` while the client was expecting an SSL connection with the server. We fix this in the PR. Release note: None Co-authored-by: souravcrl <[email protected]>

…test informs #131532 informs #131110 informs #130253 informs #127745 Epic: CRDB-41958 `TestAuthenticationAndHBARules` fails for special_cases data driven test. We suspect it might be due to client for `special_cases` test accessing the test server from a previous test `secure_non_tls` which sets `accept_sql_without_tls` to true. This results in the following error `ERROR: pq: SSL is not enabled on the server` while the client was expecting an SSL connection with the server. We fix this in the PR. Release note: None

souravcrl · 2024-10-28T09:00:32Z

@rafiss the commit for 23.1 4c0c983648a468f8f9196956662175193ae645ab had the test build PR 292e587 but the test build run logs did not have any of these added log messages: https://teamcity.cockroachdb.com/repository/download/Cockroach_Nightlies_StressBazel/17469114:id/tmp/_tmp/28616e575ae934b92eba220441c7ca42/logTestAuthenticationAndHBARules_insecure%3Dfalse_hba_default_equivalence2562826838/pgwiretest-dev.96f48ffe8e96.roach.2024-10-27T16_07_54Z.081701.log
Does this confirm that server did not explicitly send 'N' and the libpq client wrongly inferred not 'S' as 'N'? Is there something else we can do on top of this to validate?

rafiss · 2024-10-29T16:22:14Z

That is surprising. Could we try adding even more logs, like this? #133688

I wonder if this means the server never received the connection at all.

133688: pgwire: add more test logs to debug flaky test r=rafiss a=rafiss informs #127745 informs #133360 informs #131532 informs #131110 Release note: None Co-authored-by: Rafi Shamim <[email protected]>

cockroach-teamcity · 2024-11-12T21:44:41Z

sql/pgwire.TestAuthenticationAndHBARules failed with artifacts on release-23.1 @ 01f5bff3e9a800e91636444ff69b557d65153aaa:

        connect [1 args]
        <no input to command>
        ----
        ok defaultdb
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1262/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/empty_hba:45:
        connect_unix [1 args]
        <no input to command>
        ----
        ERROR: password authentication failed for user root (SQLSTATE 28P01)
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1262/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/empty_hba:51:
         
        expected:
        ERROR: password authentication failed for user root (SQLSTATE 28P01)
        
        found:
        ERROR: pq: SSL is not enabled on the server
            --- FAIL: TestAuthenticationAndHBARules/insecure=false/empty_hba/root (0.01s)
=== RUN   TestAuthenticationAndHBARules/insecure=false
    --- FAIL: TestAuthenticationAndHBARules/insecure=false (23.37s)
=== RUN   TestAuthenticationAndHBARules/insecure=false/empty_hba
    test_log_scope.go:156: test logs captured to: /artifacts/tmp/_tmp/28616e575ae934b92eba220441c7ca42/logTestAuthenticationAndHBARules_insecure=false_empty_hba2806366762
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1262/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/empty_hba:16:
        config [1 args]
        <no input to command>
        ----
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1262/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/empty_hba:20:
        set_hba [0 args]
        <no input to command>
        ----
        # Active authentication configuration on this node:
        # Original configuration:
        # host  all root all cert-password # CockroachDB mandatory rule
        # loopback all all all trust       # built-in CockroachDB default
        # host     all all all cert-password # built-in CockroachDB default
        # local    all all     password      # built-in CockroachDB default
        #
        # Interpreted configuration:
        # TYPE   DATABASE USER ADDRESS METHOD        OPTIONS
        host     all      root all     cert-password
        loopback all      all  all     trust
        host     all      all  all     cert-password
        local    all      all          password
=== CONT  TestAuthenticationAndHBARules/insecure=false/empty_hba
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/28616e575ae934b92eba220441c7ca42/logTestAuthenticationAndHBARules_insecure=false_empty_hba2806366762
        --- FAIL: TestAuthenticationAndHBARules/insecure=false/empty_hba (1.21s)

Parameters:

TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

sql/pgwire: TestAuthenticationAndHBARules failed #133360 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot T-product-security branch-release-24.3]
sql/pgwire: TestAuthenticationAndHBARules failed #132769 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot T-product-security branch-release-23.2.13-rc]
sql/pgwire: TestAuthenticationAndHBARules failed #131978 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot T-product-security branch-release-23.1.28-rc]
sql/pgwire: TestAuthenticationAndHBARules failed #131532 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-product-security branch-master]
sql/pgwire: TestAuthenticationAndHBARules failed #131110 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-product-security branch-release-23.2]
sql/pgwire: TestAuthenticationAndHBARules failed #130253 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-product-security branch-release-23.1.26-rc]

_{This test on roachdash | Improve this report!}

rafiss · 2024-11-12T22:33:42Z

The latest failure has additional debug logs present.

The last debug log we see is:

I241112 21:42:24.157507 8266 sql/pgwire/pre_serve.go:459 ⋮ [T1,n1,client=‹@›] 246  client did not request SSL version=196608 AcceptSQLWithoutTLS=false and connType=‹local›

This corresponds to the test at line 45:

cockroach/pkg/sql/pgwire/testdata/auth/empty_hba

Lines 43 to 47 in 01f5bff

    
           # However root cannot connect over the unix socket by default 
        
           # because it does not have a password. 
        
           connect_unix user=root 
        
           ---- 
        
           ERROR: password authentication failed for user root (SQLSTATE 28P01)

However, the failure is this test at line 53:

cockroach/pkg/sql/pgwire/testdata/auth/empty_hba

Lines 49 to 55 in 01f5bff

    
           # When no client cert is presented, the server would otherwise require 
        
           # password auth. However, root does not have a password. 
        
           connect user=root password=foo sslmode=verify-ca sslcert= 
        
           ---- 
        
           ERROR: password authentication failed for user root (SQLSTATE 28P01) 
        
           subtest end root

The latest logging patch was written so that we show a log for every possible error return path of the maybeUpgradeToSecureConn function. So that means we can conclude that the connection attempt never even reached CRDB. @souravcrl I believe that means we should go with the solution you proposed in #132729.

Thanks for your patience as we attempted to get to the bottom of this mystery. Another direction we can try is to switch away from the lib/pq driver (which is not maintained) and use pgx instead for this test.

souravcrl · 2024-11-14T07:14:26Z

Hey thanks for verifying this @rafiss . There are other places we use lib/pq library. Is there a need to remove from also as under stress the error handling logic is wrong?

rafiss · 2024-11-14T08:13:25Z

Unfortunately, replacing the lib/pq library everywhere in our code/tests would be an enormous project. For the past couple years we have been changing specific use cases to using pgx, but there hasn't been a need to prioritize a project to make the change across the entire codebase. Instead, the approach we use is to tactically update specific tests/packages to using pgx only when there is some need to. (For example, if there are flakes, or if a test needs a driver feature that is only in pgx.)

blathers-crl · 2024-11-14T14:22:33Z

Based on the specified backports for linked PR #135086, I applied the following new label(s) to this issue: branch-release-23.2, branch-release-24.1, branch-release-24.2, branch-release-24.3. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

cockroach-teamcity added this to the 23.1 milestone Jul 26, 2024

This was referenced Jul 28, 2024

sql/pgwire: TestAuthenticationAndHBARules failed #126239

Closed

sql/pgwire: TestAuthenticationAndHBARules failed #128094

Closed

rafiss removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Aug 5, 2024

exalate-issue-sync bot added the P-2 Issues/test failures with a fix SLA of 3 months label Aug 5, 2024

cockroach-teamcity mentioned this issue Aug 11, 2024

sql/pgwire: TestAuthenticationAndHBARules failed #128765

Closed

cockroach-teamcity mentioned this issue Aug 17, 2024

sql/pgwire: TestAuthenticationAndHBARules failed #129181

Closed

github-project-automation bot added this to SQL Foundations Aug 28, 2024

github-project-automation bot moved this to Triage in SQL Foundations Aug 28, 2024

cockroach-teamcity mentioned this issue Sep 6, 2024

sql/pgwire: TestAuthenticationAndHBARules failed #130253

Closed

cockroach-teamcity mentioned this issue Sep 20, 2024

sql/pgwire: TestAuthenticationAndHBARules failed #131110

Closed

cockroach-teamcity mentioned this issue Sep 27, 2024

sql/pgwire: TestAuthenticationAndHBARules failed #131532

Closed

souravcrl mentioned this issue Sep 30, 2024

sql: fix TestAuthenticationAndHBARules for special_cases data-driven test #131580

Merged

exalate-issue-sync bot added T-product-security and removed T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) labels Sep 30, 2024

blathers-crl bot mentioned this issue Oct 1, 2024

release-23.1: sql: fix TestAuthenticationAndHBARules for special_cases data-driven test #131650

Merged

blathers-crl bot mentioned this issue Oct 1, 2024

release-23.2: sql: fix TestAuthenticationAndHBARules for special_cases data-driven test #131651

Merged

exalate-issue-sync bot assigned souravcrl Oct 28, 2024

rafiss mentioned this issue Oct 29, 2024

pgwire: add more test logs to debug flaky test #133688

Merged

craig bot pushed a commit that referenced this issue Oct 29, 2024

Merge #133688

f759a46

133688: pgwire: add more test logs to debug flaky test r=rafiss a=rafiss informs #127745 informs #133360 informs #131532 informs #131110 Release note: None Co-authored-by: Rafi Shamim <[email protected]>

rafiss mentioned this issue Nov 13, 2024

pgwire,authccl: use pgx for TestAuthenticationAndHBARules #135086

Merged

exalate-issue-sync bot removed the P-2 Issues/test failures with a fix SLA of 3 months label Nov 13, 2024

craig bot closed this as completed in afd5d06 Nov 14, 2024

blathers-crl bot mentioned this issue Nov 14, 2024

release-24.3: pgwire,authccl: use pgx for TestAuthenticationAndHBARules #135165

Merged

This was referenced Dec 19, 2024

[Replicated] release-23.1: pgwire,authccl: use pgx for TestAuthenticationAndHBARules mohini-crl/cockroach#34

Merged

[Replicated] release-23.1: pgwire: add more test logs to debug flaky test mohini-crl/cockroach#66

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sql/pgwire: TestAuthenticationAndHBARules failed #127745

sql/pgwire: TestAuthenticationAndHBARules failed #127745

cockroach-teamcity commented Jul 26, 2024 •

edited by cockroach-jira-scripts

Loading

cockroach-teamcity commented Aug 14, 2024

cockroach-teamcity commented Aug 28, 2024

cockroach-teamcity commented Sep 10, 2024

souravcrl commented Oct 28, 2024

rafiss commented Oct 29, 2024

cockroach-teamcity commented Nov 12, 2024

rafiss commented Nov 12, 2024

souravcrl commented Nov 14, 2024

rafiss commented Nov 14, 2024

blathers-crl bot commented Nov 14, 2024

sql/pgwire: TestAuthenticationAndHBARules failed #127745

sql/pgwire: TestAuthenticationAndHBARules failed #127745

Comments

cockroach-teamcity commented Jul 26, 2024 • edited by cockroach-jira-scripts Loading

cockroach-teamcity commented Aug 14, 2024

cockroach-teamcity commented Aug 28, 2024

cockroach-teamcity commented Sep 10, 2024

souravcrl commented Oct 28, 2024

rafiss commented Oct 29, 2024

cockroach-teamcity commented Nov 12, 2024

rafiss commented Nov 12, 2024

souravcrl commented Nov 14, 2024

rafiss commented Nov 14, 2024

blathers-crl bot commented Nov 14, 2024

cockroach-teamcity commented Jul 26, 2024 •

edited by cockroach-jira-scripts

Loading