roachtest: add node-kill operation

[itsbilal]

This change adds a node-kill operation with 4 variants: one that drains and one that doesn't x {SIGKILL, SIGTERM}.

Epic: none

Release note: None

[cockroach-teamcity]

This change is 

[herkolategan]

Left a few small comments, mostly roachtest convenience APIs related. One general question, It seems that clean-up is responsible for the timing regarding when the node is started back up again. Would there be value in having nodes be down for a specified (random) amount of time, rather than a fixed amount of time?

[herkolategan]

@renatolabs may know how to structure this better, but we could instead of doing an ExternalPGUrl use the new command util to optionally add insecure or certs to drain:

Something like this (untested):

cmd := roachtestutil.NewCommand(fmt.Sprintf("%s node drain", test.DefaultCockroachPath)).
	Flag("logtostderr", "INFO").
	Flag("port", fmt.Sprintf("{pgport:%d}", nid)).
	MaybeOption(!c.IsSecure(),"insecure").
	MaybeFlag(c.IsSecure(),"certs", install.CockroachNodeCertsDir).
	Option("self")
err = c.RunE(ctx, option.WithNodes(c.Node(nid)), cmd.String())

Also not sure if secure will be correct here, it's normally passed in (stateless)

[itsbilal]

Thanks for the review! Addressed the points brought up.

Would there be value in having nodes be down for a specified (random) amount of time, rather than a fixed amount of time?

This is a good point, and something I'd want drt-run (which I have been working on again lately) to handle. Currently run-operation --wait-before-cleanup <duration> configures it. Because yes, we'd ideally want a more interesting interleaving of operation-induced states.

[itsbilal]

TFTR!

bors r=renatolabs

[craig]

This PR was included in a batch that was canceled, it will be automatically retried

[craig]

Build failed (retrying...):

• lint

[yuzefovich]

The linter is not happy.

bors r-

[craig]

Canceled.

[itsbilal]

Apologies, didn't realize the linter isn't under Bazel essential CI anymore.

[itsbilal]

Required checks are green now.

bors r=renatolabs

[craig]

Build succeeded:

• linux_amd64_build
• lint
• acceptance
• macos_amd64_build
• linux_amd64_fips_build
• macos_arm64_build
• local_roachtest_fips
• docker_image_amd64
• unit_tests
• check_generated_code
• linux_arm64_build
• local_roachtest
• examples_orms
• windows_build