pkg/server: TestDecommissionSelf failed

[cockroach-teamcity]

pkg/server/storage_api/storage_api_test.TestDecommissionSelf failed with artifacts on master @ 3033423ccce8b4167dc7621d66d069ad003c1eb5:

=== RUN   TestDecommissionSelf
    test_log_scope.go:167: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestDecommissionSelf708434112
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:104: 
        Test server was configured to route SQL queries to a secondary tenant (virtual cluster).
        If you are only seeing a test failure when this message appears, there may be a problem
        specific to cluster virtualization or multi-tenancy.
        
        To investigate, consider using "COCKROACH_TEST_TENANT=true" to force-enable just
        the secondary tenant in all runs (or, alternatively, "false" to force-disable), or use
        "COCKROACH_INTERNAL_DISABLE_METAMORPHIC_TESTING=false" to disable all random test variables altogether.
*
* ERROR: test tenant requested by configuration, but code organization prevents start!
* OSS binaries do not include enterprise features
*
*
* ERROR: test tenant requested by configuration, but code organization prevents start!
* OSS binaries do not include enterprise features
*
*
* ERROR: test tenant requested by configuration, but code organization prevents start!
* OSS binaries do not include enterprise features
*
*
* ERROR: test tenant requested by configuration, but code organization prevents start!
* OSS binaries do not include enterprise features
*
*
* ERROR: test tenant requested by configuration, but code organization prevents start!
* OSS binaries do not include enterprise features
*
*
* ERROR: test tenant requested by configuration, but code organization prevents start!
* OSS binaries do not include enterprise features
*
*
* ERROR: test tenant requested by configuration, but code organization prevents start!
* OSS binaries do not include enterprise features
*
    decommission_test.go:698: 
        	Error Trace:	pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:698
        	Error:      	Received unexpected error:
        	            	rpc error: code = Internal desc = failed to connect to n1 at 127.0.0.1:46583: grpc: n5 was permanently removed from the cluster at 2023-07-20 06:19:53.131281476 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned
        	Test:       	TestDecommissionSelf
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestDecommissionSelf708434112
--- FAIL: TestDecommissionSelf (4.30s)

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/unowned

This test on roachdash | Improve this report!

Jira issue: CRDB-29956

[blathers-crl]

cc @cockroachdb/replication

[github-actions]

We have marked this test failure issue as stale because it has been
inactive for 1 month. If this failure is still relevant, removing the
stale label or adding a comment will keep it active. Otherwise,
we'll close it in 5 days to keep the test failure queue tidy.

[AlexTalks]

Could not reproduce this issue. May have been fixed by #108762, but even removing the code opting out of running the test under a secondary tenant and enabling it explicitly with COCKROACH_TEST_TENANT=true, I wasn't able to reproduce this issue while stressing on either Linux or MacOS.