Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TestRunServer/SecureTenant is broken with CockroachDB master #87

Closed
rafiss opened this issue Aug 6, 2020 · 3 comments
Closed

TestRunServer/SecureTenant is broken with CockroachDB master #87

rafiss opened this issue Aug 6, 2020 · 3 comments
Assignees
@tbg

Comments

@rafiss
Copy link
Contributor

rafiss commented Aug 6, 2020

This test is currently broken:

=== RUN   TestRunServer
=== RUN   TestRunServer/SecureTenant
2020/08/06 16:17:51 Using custom cockroach binary: /Users/rafiss/go/src/github.com/cockroachdb/cockroach/cockroach
2020/08/06 16:17:51 process 95859 started: /Users/rafiss/go/src/github.com/cockroachdb/cockroach/cockroach start-single-node --logtostderr --certs-dir=/tmp/cockroach-testserver274475142/certs --host=localhost --port=0 --http-port=0 --store=/tmp/cockroach-testserver274475142 --listening-url-file=/tmp/cockroach-testserver274475142/listen-url
2020/08/06 16:17:53 process 95886 started: /Users/rafiss/go/src/github.com/cockroachdb/cockroach/cockroach mt start-sql --certs-dir=/tmp/cockroach-testserver274475142/certs --logtostderr --tenant-id=2 --kv-addrs=localhost:50535 --sql-addr=localhost:50545
2020/08/06 16:17:53 WaitForInit: Trying again after error: dial tcp [::1]:50545: connect: connection refused
2020/08/06 16:17:53 Process 95886 exited with status 2: exit status 2
2020/08/06 16:17:53 exit status 2
2020/08/06 16:17:53 WaitForInit: Trying again after error: dial tcp [::1]:50545: connect: connection refused
2020/08/06 16:17:53 WaitForInit: Trying again after error: dial tcp [::1]:50545: connect: connection refused
2020/08/06 16:17:53 WaitForInit: Trying again after error: dial tcp [::1]:50545: connect: connection refused
2020/08/06 16:17:53 WaitForInit: Trying again after error: dial tcp [::1]:50545: connect: connection refused
...

--- FAIL: TestRunServer (7.58s)
    testserver_test.go:44: dial tcp [::1]:50545: connect: connection refused
    --- FAIL: TestRunServer/SecureTenant (7.58s)

I was able to reproduce locally, and see this error when starting the tenant:

❯ /Users/rafiss/go/src/github.com/cockroachdb/cockroach/cockroach mt start-sql --certs-dir=/tmp/cockroach-testserver274475142/certs --logtostderr --tenant-id=2 --kv-addrs=localhost:50535 --sql-addr=localhost:50545
I200806 20:18:51.231010 15 util/stop/stopper.go:563  quiescing
E200806 20:18:51.230998 1 util/log/crash_reporting.go:138  a panic has occurred!
for 2: tenant client cert not found
(1) attached stack trace
  | runtime.gopanic
  | 	/usr/local/Cellar/go/1.14.1/libexec/src/runtime/panic.go:967
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop
  | 	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:503
  | runtime.gopanic
  | 	/usr/local/Cellar/go/1.14.1/libexec/src/runtime/panic.go:967
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServer
  | 	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:216
  | github.com/cockroachdb/cockroach/pkg/server.makeSQLServerArgs
  | 	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:561
  | github.com/cockroachdb/cockroach/pkg/server.StartTenant
  | 	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:641
  | github.com/cockroachdb/cockroach/pkg/cli.runStartSQL
  | 	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/cli/mt_start_sql.go:68
  | github.com/cockroachdb/cockroach/pkg/cli.MaybeDecorateGRPCError.func1
  | 	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/cli/error.go:189
  | github.com/spf13/cobra.(*Command).execute
  | 	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:826
  | github.com/spf13/cobra.(*Command).ExecuteC
  | 	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
  | github.com/spf13/cobra.(*Command).Execute
  | 	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
  | github.com/cockroachdb/cockroach/pkg/cli.Run
  | 	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:225
  | github.com/cockroachdb/cockroach/pkg/cli.Main
  | 	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:67
  | main.main
  | 	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/cmd/cockroach-short/main.go:26
  | runtime.main
  | 	/usr/local/Cellar/go/1.14.1/libexec/src/runtime/proc.go:203
  | runtime.goexit
  | 	/usr/local/Cellar/go/1.14.1/libexec/src/runtime/asm_amd64.s:1373
Wraps: (2) for 2: tenant client cert not found
Error types: (1) *withstack.withStack (2) *security.Error
E200806 20:18:51.231616 1 util/log/crash_reporting.go:138  a panic has occurred!
for 2: tenant client cert not found
(1) attached stack trace
  | runtime.gopanic
  | 	/usr/local/Cellar/go/1.14.1/libexec/src/runtime/panic.go:967
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop
  | 	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:503
  | runtime.gopanic
  | 	/usr/local/Cellar/go/1.14.1/libexec/src/runtime/panic.go:967
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServer
  | 	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:216
  | github.com/cockroachdb/cockroach/pkg/server.makeSQLServerArgs
  | 	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:561
  | github.com/cockroachdb/cockroach/pkg/server.StartTenant
  | 	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:641
  | github.com/cockroachdb/cockroach/pkg/cli.runStartSQL
  | 	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/cli/mt_start_sql.go:68
  | github.com/cockroachdb/cockroach/pkg/cli.MaybeDecorateGRPCError.func1
  | 	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/cli/error.go:189
  | github.com/spf13/cobra.(*Command).execute
  | 	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:826
  | github.com/spf13/cobra.(*Command).ExecuteC
  | 	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
  | github.com/spf13/cobra.(*Command).Execute
  | 	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
  | github.com/cockroachdb/cockroach/pkg/cli.Run
  | 	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:225
  | github.com/cockroachdb/cockroach/pkg/cli.Main
  | 	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:67
  | main.main
  | 	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/cmd/cockroach-short/main.go:26
  | runtime.main
  | 	/usr/local/Cellar/go/1.14.1/libexec/src/runtime/proc.go:203
  | runtime.goexit
  | 	/usr/local/Cellar/go/1.14.1/libexec/src/runtime/asm_amd64.s:1373
Wraps: (2) for 2: tenant client cert not found
Error types: (1) *withstack.withStack (2) *security.Error
panic: for 2: tenant client cert not found [recovered]
	panic: for 2: tenant client cert not found [recovered]
	panic: for 2: tenant client cert not found [recovered]
	panic: for 2: tenant client cert not found

goroutine 1 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.RecoverAndReportPanic(0x9091a60, 0xc00007a160, 0xc000016a80)
	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/util/log/crash_reporting.go:88 +0x9f
panic(0x7e15f20, 0xc000da45c0)
	/usr/local/Cellar/go/1.14.1/libexec/src/runtime/panic.go:967 +0x166
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc00011fdd0, 0x9091a60, 0xc00007a160)
	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:207 +0x11f
panic(0x7e15f20, 0xc000da45c0)
	/usr/local/Cellar/go/1.14.1/libexec/src/runtime/panic.go:967 +0x166
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop(0xc00011fdd0, 0x9091a60, 0xc00007a160)
	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:503 +0x68c
panic(0x7e15f20, 0xc000da45c0)
	/usr/local/Cellar/go/1.14.1/libexec/src/runtime/panic.go:967 +0x166
github.com/cockroachdb/cockroach/pkg/rpc.NewServer(0xc0000f6240, 0x0, 0x0, 0x0, 0x90db8a0)
	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:216 +0xe7c
github.com/cockroachdb/cockroach/pkg/server.makeSQLServerArgs(0xc00011fdd0, 0x0, 0x0, 0xc000016a80, 0xc000197400, 0x906e860, 0xc000adf8a0, 0x0, 0x0, 0x0, ...)
	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:561 +0xbda
github.com/cockroachdb/cockroach/pkg/server.StartTenant(0x9091a60, 0xc00007a160, 0xc00011fdd0, 0x0, 0x0, 0xc000016a80, 0xc000197400, 0x906e860, 0xc000adf8a0, 0x0, ...)
	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:641 +0xf8
github.com/cockroachdb/cockroach/pkg/cli.runStartSQL(0xb7685c0, 0xc000b0b450, 0x0, 0x5, 0x0, 0x0)
	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/cli/mt_start_sql.go:68 +0x2c4
github.com/cockroachdb/cockroach/pkg/cli.MaybeDecorateGRPCError.func1(0xb7685c0, 0xc000b0b450, 0x0, 0x5, 0x0, 0x0)
	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/cli/error.go:189 +0x79
github.com/spf13/cobra.(*Command).execute(0xb7685c0, 0xc000b0b400, 0x5, 0x5, 0xb7685c0, 0xc000b0b400)
	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:826 +0x453
github.com/spf13/cobra.(*Command).ExecuteC(0xb75f740, 0xc00007a160, 0xc00062b62a, 0xc)
	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
github.com/cockroachdb/cockroach/pkg/cli.Run(...)
	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:225
github.com/cockroachdb/cockroach/pkg/cli.Main()
	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:67 +0x270
main.main()
	/Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/cmd/cockroach-short/main.go:26 +0x20
@rafiss rafiss mentioned this issue Aug 6, 2020
Merged
@rafiss
Copy link
Contributor Author

rafiss commented Aug 6, 2020

@asubiotto or @tbg would one of you be able to look into this?

I see that this landed recently: cockroachdb/cockroach#51503 Could it be related?

@rafiss rafiss mentioned this issue Aug 6, 2020
Merged
@tbg tbg self-assigned this Aug 7, 2020
@tbg
Copy link
Member

tbg commented Aug 7, 2020

I'll take a look!

@tbg
Copy link
Member

tbg commented Aug 7, 2020

@darinpp and I are just about to simplify the certs story here quite a bit, so I'm going to hold off on fixing this test for a moment or two. We will need to touch this anway to add in the SQL proxy in cockroachdb/cockroach#52478.

@tbg tbg mentioned this issue Aug 18, 2020
Closed
@tbg tbg closed this as completed in 23be265 Aug 19, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tbg tbg

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

testserver: add missing tenant client certificate cockroachdb/cockroach-go
2 participants
@rafiss @tbg