sql/pgwire: TestAuthenticationAndHBARules failed

[cockroach-teamcity]

sql/pgwire.TestAuthenticationAndHBARules failed with artifacts on master @ cafd11fa98b93cdaf43f4d0210947fb2c0fbcd25:

=== RUN   TestAuthenticationAndHBARules
--- FAIL: TestAuthenticationAndHBARules (32.08s)
=== RUN   TestAuthenticationAndHBARules/insecure=false
    --- FAIL: TestAuthenticationAndHBARules/insecure=false (20.87s)
=== RUN   TestAuthenticationAndHBARules/insecure=false/special_cases
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/9df7e2b43c6703c48b37b40add1a12cd/logTestAuthenticationAndHBARules_insecure=false_special_cases3799389318
    test_server_shim.go:93: cluster virtualization disabled due to issue: #107310 (expected label: C-bug)
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/14828/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/special_cases:3:
        config [1 args]
        <no input to command>
        ----
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/14828/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/special_cases:6:
        sql [0 args]
        CREATE USER passworduser WITH PASSWORD 'pass'
        ----
        ok
    panic.go:626: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/9df7e2b43c6703c48b37b40add1a12cd/logTestAuthenticationAndHBARules_insecure=false_special_cases3799389318
        --- FAIL: TestAuthenticationAndHBARules/insecure=false/special_cases (0.70s)
=== RUN   TestAuthenticationAndHBARules/insecure=false/special_cases/root_user_cannot_use_password
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/14828/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/special_cases:17:
        set_hba [0 args]
        host all root 0.0.0.0/0 password
        ----
        # Active authentication configuration on this node:
        # Original configuration:
        # loopback all all all trust       # built-in CockroachDB default
        # host  all root all cert-password # CockroachDB mandatory rule
        # host all root 0.0.0.0/0 password
        #
        # Interpreted configuration:
        # TYPE   DATABASE USER ADDRESS   METHOD        OPTIONS
        loopback all      all  all       trust
        host     all      root all       cert-password
        host     all      root 0.0.0.0/0 password
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/14828/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/special_cases:32:
         
        expected:
        ERROR: password authentication failed for user root (SQLSTATE 28P01)
        
        found:
        ERROR: pq: SSL is not enabled on the server
            --- FAIL: TestAuthenticationAndHBARules/insecure=false/special_cases/root_user_cannot_use_password (0.02s)

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

• sql/pgwire: TestAuthenticationAndHBARules failed #131110 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-23.2]
• sql/pgwire: TestAuthenticationAndHBARules failed #130253 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-23.1.26-rc]
• sql/pgwire: TestAuthenticationAndHBARules failed #127745 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-23.1]

/cc @cockroachdb/sql-foundations @cockroachdb/server @cockroachdb/product-security

This test on roachdash | Improve this report!

Jira issue: CRDB-42588

[souravcrl]

@yuzefovich This seems to be a flake as I can't reproduce this locally. This might be related to test tenants bug which prevents TestAuthenticationAndHBARules from running in multi tenant mode #107310.
As mentioned in issue, this might be related to AcceptSQLWithoutTLS which gets set in another data driven test.
I have a PR out which addresses this #131580.
cc: @pritesh-lahoti

[souravcrl]

I am also seeing the log is server logs

I240927 18:43:12.286698 42625 1@util/log/event_log.go:44 ⋮ [T1,Vsystem,n1,client=127.0.0.1:41266,hostssl,user=root] 367 ={"Timestamp":1727462592284106044,"EventType":"set_cluster_setting","Statement":"SET CLUSTER SETTING \"server.host_based_authentication.configuration\" = $1","Tag":"SET CLUSTER SETTING","User":"root","PlaceholderValues":["‹'host all root 0.0.0.0/0 password'›"],"SettingName":"server.host_based_authentication.configuration","Value":"‹'host all root 0.0.0.0/0 password'›"}
W240927 18:43:12.299470 42884 kv/kvserver/intentresolver/intent_resolver.go:862 ⋮ [-] 368  failed to gc transaction record: could not GC completed transaction anchored at /Table/4/1/‹"passworduser"›/‹2›/‹1›: node unavailable; try another peer

The message says could not GC completed transaction anchored at /Table/4/1/‹"passworduser"›/‹2›/‹1›: node unavailable; try another peer
So might be related to kv failure

[rafiss]

@souravcrl in case you hadn't seen, this is some information I found when I last investigated this test: #126239 (comment)

[souravcrl]

@rafiss had missed that. Thanks for the pointer. I am also not sure how this case could be since we start the server in secure mode for the test special_cases and GetServerTLSConfig also does not return an error implying server was started in insecure mode, otherwise we would have seen a certs error:

cockroach/pkg/rpc/tls.go

Lines 147 to 166 in 607c697

	// GetServerTLSConfig returns the server TLS config, initializing it if needed.
	// If Insecure is true, return a nil config, otherwise ask the certificate
	// manager for a server TLS config.
	func (ctx *SecurityContext) GetServerTLSConfig() (*tls.Config, error) {
	// Early out.
	if ctx.config.Insecure {
	return nil, nil
	}
	cm, err := ctx.GetCertificateManager()
	if err != nil {
	return nil, wrapError(err)
	}
	tlsCfg, err := cm.GetServerTLSConfig()
	if err != nil {
	return nil, wrapError(err)
	}
	return tlsCfg, nil
	}

.
It is possible that the goroutine running the client for the test TestAuthenticationAndHBARules/insecure=false had a leak and is accessing the test server setup for TestAuthenticationAndHBARules/insecure=true/....

[blathers-crl]

Based on the specified backports for linked PR #135086, I applied the following new label(s) to this issue: branch-release-23.1, branch-release-23.2, branch-release-24.1, branch-release-24.2, branch-release-24.3. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}