Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: cluster_creation failed #78601

Closed
cockroach-teamcity opened this issue Mar 28, 2022 · 633 comments · Fixed by #105234
Closed

roachtest: cluster_creation failed #78601

cockroach-teamcity opened this issue Mar 28, 2022 · 633 comments · Fixed by #105234
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. sync-me-8 T-testeng TestEng Team

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Mar 28, 2022 •
edited by cockroach-jira-scripts
Loading

roachtest.cluster_creation failed with artifacts on master @ 32b45c4bcf1ab41f0ba3abd36cb670eea7f450fd:

The test failed on branch=master, cloud=aws:
test tpccbench/nodes=3/cpu=16 was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c5d.4xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-K9sxDtvNHA4eytNlf-coAmkqWJs= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4698278-1648444789-10-n4cpu16},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-03-28T12:48:24Z},{Key=Name,Value=teamcity-4698278-1648444789-10-n4cpu16-0002}] --user-data file:///tmp/aws-startup-script3706005379 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping4036371810 --output json: stderr: : exit status 254
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: aws
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runCommand
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:194
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runJSONCommand
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:204
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runInstance
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:969
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).Create.func1
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:443
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c5d.4xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-K9sxDtvNHA4eytNlf-coAmkqWJs= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4698278-1648444789-10-n4cpu16},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-03-28T12:48:24Z},{Key=Name,Value=teamcity-4698278-1648444789-10-n4cpu16-0002}] --user-data file:///tmp/aws-startup-script3706005379 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping4036371810 --output json: stderr: 
		  | An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c5d.4xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c5d.4xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.
		Wraps: (5) exit status 254
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

/cc @cockroachdb/dev-inf

This test on roachdash | Improve this report!

Jira issue: CRDB-14193
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Mar 28, 2022
@cockroach-teamcity cockroach-teamcity mentioned this issue Mar 28, 2022
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 32b45c4bcf1ab41f0ba3abd36cb670eea7f450fd:

The test failed on branch=master, cloud=aws:
test tpccbench/nodes=3/cpu=16/no-admission was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c5d.4xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-K9sxDtvNHA4eytNlf-coAmkqWJs= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Name,Value=teamcity-4698278-1648444789-13-n4cpu16-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4698278-1648444789-13-n4cpu16},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-03-28T12:49:27Z}] --user-data file:///tmp/aws-startup-script3564196625 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3289024918 --output json: stderr: : exit status 254
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: aws
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runCommand
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:194
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runJSONCommand
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:204
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runInstance
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:969
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).Create.func1
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:443
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c5d.4xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-K9sxDtvNHA4eytNlf-coAmkqWJs= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Name,Value=teamcity-4698278-1648444789-13-n4cpu16-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4698278-1648444789-13-n4cpu16},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-03-28T12:49:27Z}] --user-data file:///tmp/aws-startup-script3564196625 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3289024918 --output json: stderr: 
		  | An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c5d.4xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c5d.4xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.
		Wraps: (5) exit status 254
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@jlinder jlinder added T-testeng TestEng Team and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Mar 28, 2022
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 32b45c4bcf1ab41f0ba3abd36cb670eea7f450fd:

The test failed on branch=master, cloud=aws:
test kv0/enc=false/nodes=3/cpu=96 was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-K9sxDtvNHA4eytNlf-coAmkqWJs= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Created,Value=2022-03-28T15:20:57Z},{Key=Name,Value=teamcity-4698278-1648444789-17-n4cpu96-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4698278-1648444789-17-n4cpu96},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script4078450040 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3290289358 --output json: stderr: : exit status 254
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: aws
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runCommand
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:194
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runJSONCommand
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:204
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runInstance
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:969
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).Create.func1
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:443
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-K9sxDtvNHA4eytNlf-coAmkqWJs= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Created,Value=2022-03-28T15:20:57Z},{Key=Name,Value=teamcity-4698278-1648444789-17-n4cpu96-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4698278-1648444789-17-n4cpu96},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script4078450040 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3290289358 --output json: stderr: 
		  | An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m5d.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get m5d.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.
		Wraps: (5) exit status 254
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 32b45c4bcf1ab41f0ba3abd36cb670eea7f450fd:

The test failed on branch=master, cloud=aws:
test kv95/enc=false/nodes=3/cpu=96 was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-K9sxDtvNHA4eytNlf-coAmkqWJs= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4698278-1648444789-20-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-03-28T15:21:54Z},{Key=Name,Value=teamcity-4698278-1648444789-20-n4cpu96-0002}] --user-data file:///tmp/aws-startup-script1956457138 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping389602635 --output json: stderr: : exit status 254
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: aws
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runCommand
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:194
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runJSONCommand
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:204
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runInstance
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:969
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).Create.func1
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:443
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-K9sxDtvNHA4eytNlf-coAmkqWJs= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4698278-1648444789-20-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-03-28T15:21:54Z},{Key=Name,Value=teamcity-4698278-1648444789-20-n4cpu96-0002}] --user-data file:///tmp/aws-startup-script1956457138 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping389602635 --output json: stderr: 
		  | An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m5d.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get m5d.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.
		Wraps: (5) exit status 254
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 0e6878b6140cb5cc7c799fdf576b3d565a54fd2e:

The test failed on branch=master, cloud=aws:
test kv0/enc=false/nodes=3/cpu=96 was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-iVwxzIh71NgwXPrw6hLVgSGQqB0= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4766499-1648876713-12-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-04-02T18:15:37Z},{Key=Name,Value=teamcity-4766499-1648876713-12-n4cpu96-0001}] --user-data file:///tmp/aws-startup-script3560179776 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping360008570 --output json: stderr: : exit status 254
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: aws
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runCommand
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:194
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runJSONCommand
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:204
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runInstance
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:969
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).Create.func1
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:443
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-iVwxzIh71NgwXPrw6hLVgSGQqB0= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4766499-1648876713-12-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-04-02T18:15:37Z},{Key=Name,Value=teamcity-4766499-1648876713-12-n4cpu96-0001}] --user-data file:///tmp/aws-startup-script3560179776 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping360008570 --output json: stderr: 
		  | An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m5d.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get m5d.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.
		Wraps: (5) exit status 254
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 0e6878b6140cb5cc7c799fdf576b3d565a54fd2e:

The test failed on branch=master, cloud=aws:
test kv95/enc=false/nodes=3/cpu=96 was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-iVwxzIh71NgwXPrw6hLVgSGQqB0= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Name,Value=teamcity-4766499-1648876713-15-n4cpu96-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4766499-1648876713-15-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-04-02T18:16:35Z}] --user-data file:///tmp/aws-startup-script309430335 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1555857328 --output json: stderr: : exit status 254
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: aws
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runCommand
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:194
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runJSONCommand
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:204
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runInstance
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:969
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).Create.func1
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:443
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-iVwxzIh71NgwXPrw6hLVgSGQqB0= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Name,Value=teamcity-4766499-1648876713-15-n4cpu96-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4766499-1648876713-15-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-04-02T18:16:35Z}] --user-data file:///tmp/aws-startup-script309430335 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1555857328 --output json: stderr: 
		  | An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m5d.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get m5d.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.
		Wraps: (5) exit status 254
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 72b71b6920fe5b6dde2c7229b4edc2f95432839e:

The test failed on branch=master, cloud=aws:
test kv0/enc=false/nodes=3/cpu=96 was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-vH_ZEImoEFvuuLwGxire-qhkHPc= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4807858-1649222398-11-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-04-06T18:03:52Z},{Key=Name,Value=teamcity-4807858-1649222398-11-n4cpu96-0003}] --user-data file:///tmp/aws-startup-script3252835754 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2609775494 --output json: stderr: : exit status 254
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: aws
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runCommand
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:194
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runJSONCommand
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:204
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runInstance
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:969
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).Create.func1
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:443
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-vH_ZEImoEFvuuLwGxire-qhkHPc= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4807858-1649222398-11-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-04-06T18:03:52Z},{Key=Name,Value=teamcity-4807858-1649222398-11-n4cpu96-0003}] --user-data file:///tmp/aws-startup-script3252835754 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2609775494 --output json: stderr: 
		  | An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m5d.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get m5d.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.
		Wraps: (5) exit status 254
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 72b71b6920fe5b6dde2c7229b4edc2f95432839e:

The test failed on branch=master, cloud=aws:
test kv95/enc=false/nodes=3/cpu=96 was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-vH_ZEImoEFvuuLwGxire-qhkHPc= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Created,Value=2022-04-06T18:04:49Z},{Key=Name,Value=teamcity-4807858-1649222398-14-n4cpu96-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4807858-1649222398-14-n4cpu96},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script180643413 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1999646734 --output json: stderr: : exit status 254
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: aws
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runCommand
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:194
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runJSONCommand
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:204
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runInstance
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:969
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).Create.func1
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:443
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-vH_ZEImoEFvuuLwGxire-qhkHPc= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Created,Value=2022-04-06T18:04:49Z},{Key=Name,Value=teamcity-4807858-1649222398-14-n4cpu96-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4807858-1649222398-14-n4cpu96},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script180643413 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1999646734 --output json: stderr: 
		  | An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m5d.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get m5d.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.
		Wraps: (5) exit status 254
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 7a97bdd02c62787f40d79d5a2cd3439d8c9c8a1b:

The test failed on branch=master, cloud=gce:
test kv0/enc=false/nodes=3/cpu=96 was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-96 --labels roachprod=true,cluster=teamcity-4831888-1649395225-70-n4cpu96,lifetime=12h0m0s,created=2022-04-08t10_51_00z --metadata-from-file startup-script=/tmp/gce-startup-script3623791391 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-96 --labels roachprod=true,cluster=teamcity-4831888-1649395225-70-n4cpu96,lifetime=12h0m0s,created=2022-04-08t10_51_00z --metadata-from-file startup-script=/tmp/gce-startup-script3623791391 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4831888-1649395225-70-n4cpu96-0001].
		  | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4831888-1649395225-70-n4cpu96-0002].
		  | WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  '(resource type:compute)'.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 7a97bdd02c62787f40d79d5a2cd3439d8c9c8a1b:

The test failed on branch=master, cloud=gce:
test kv50/enc=false/nodes=4/cpu=96/batch=64 was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-96 --labels cluster=teamcity-4831888-1649395225-100-n5cpu96,lifetime=12h0m0s,created=2022-04-08t15_01_29z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script595119232 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-96 --labels cluster=teamcity-4831888-1649395225-100-n5cpu96,lifetime=12h0m0s,created=2022-04-08t15_01_29z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script595119232 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4831888-1649395225-100-n5cpu96-0001].
		  | WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  '(resource type:compute)'.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test alterpk-tpcc-500 was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897747-1649999913-23-n4cpu16,lifetime=12h0m0s,created=2022-04-15t05_51_05z --metadata-from-file startup-script=/tmp/gce-startup-script1162684711 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897747-1649999913-23-n4cpu16,lifetime=12h0m0s,created=2022-04-15t05_51_05z --metadata-from-file startup-script=/tmp/gce-startup-script1162684711 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test asyncpg was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897747-1649999913-25-n1cpu16,lifetime=12h0m0s,created=2022-04-15t05_51_22z --metadata-from-file startup-script=/tmp/gce-startup-script1289394172 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897747-1649999913-25-n1cpu16,lifetime=12h0m0s,created=2022-04-15t05_51_22z --metadata-from-file startup-script=/tmp/gce-startup-script1289394172 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test autoupgrade was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-4 --labels lifetime=12h0m0s,created=2022-04-15t05_51_52z,roachprod=true,cluster=teamcity-4897747-1649999913-31-n5cpu4 --metadata-from-file startup-script=/tmp/gce-startup-script1727293541 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-4 --labels lifetime=12h0m0s,created=2022-04-15t05_51_52z,roachprod=true,cluster=teamcity-4897747-1649999913-31-n5cpu4 --metadata-from-file startup-script=/tmp/gce-startup-script1727293541 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test backup/2TB/n10cpu4 was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-4 --labels cluster=teamcity-4897747-1649999913-37-n10cpu4,lifetime=12h0m0s,created=2022-04-15t05_52_24z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script2921408233 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-4 --labels cluster=teamcity-4897747-1649999913-37-n10cpu4,lifetime=12h0m0s,created=2022-04-15t05_52_24z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script2921408233 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test backup/KMS/AWS/n3cpu4 was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-38-n3cpu4,lifetime=12h0m0s,created=2022-04-15t05_52_25z --metadata-from-file startup-script=/tmp/gce-startup-script337372255 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-38-n3cpu4,lifetime=12h0m0s,created=2022-04-15t05_52_25z --metadata-from-file startup-script=/tmp/gce-startup-script337372255 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test backupTPCC was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-40-n3cpu4,lifetime=12h0m0s,created=2022-04-15t05_52_37z --metadata-from-file startup-script=/tmp/gce-startup-script2392853458 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-40-n3cpu4,lifetime=12h0m0s,created=2022-04-15t05_52_37z --metadata-from-file startup-script=/tmp/gce-startup-script2392853458 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Apr 15, 2022
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test cdc/crdb-chaos was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels cluster=teamcity-4897747-1649999913-48-n4cpu16,lifetime=12h0m0s,created=2022-04-15t05_53_48z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script1777832219 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels cluster=teamcity-4897747-1649999913-48-n4cpu16,lifetime=12h0m0s,created=2022-04-15t05_53_48z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script1777832219 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test cdc/initial-scan was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897747-1649999913-53-n4cpu16,lifetime=12h0m0s,created=2022-04-15t05_54_57z --metadata-from-file startup-script=/tmp/gce-startup-script2645437982 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897747-1649999913-53-n4cpu16,lifetime=12h0m0s,created=2022-04-15t05_54_57z --metadata-from-file startup-script=/tmp/gce-startup-script2645437982 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test cdc/cloud-sink-gcs/rangefeed=true was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels created=2022-04-15t05_53_36z,roachprod=true,cluster=teamcity-4897747-1649999913-47-n4cpu16,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script87802300 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels created=2022-04-15t05_53_36z,roachprod=true,cluster=teamcity-4897747-1649999913-47-n4cpu16,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script87802300 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-47-n4cpu16-0004].
		  | WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test cdc/ledger was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels cluster=teamcity-4897747-1649999913-60-n4cpu16,lifetime=12h0m0s,created=2022-04-15t05_58_26z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script3173295066 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels cluster=teamcity-4897747-1649999913-60-n4cpu16,lifetime=12h0m0s,created=2022-04-15t05_58_26z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script3173295066 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-60-n4cpu16-0001].
		  | WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test cdc/tpcc-1000 was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897747-1649999913-64-n4cpu16,lifetime=12h0m0s,created=2022-04-15t06_01_51z --metadata-from-file startup-script=/tmp/gce-startup-script496068105 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897747-1649999913-64-n4cpu16,lifetime=12h0m0s,created=2022-04-15t06_01_51z --metadata-from-file startup-script=/tmp/gce-startup-script496068105 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 13addf52fe9b979f5cc3aeabac518058ec7715ff:

test ycsb/E/nodes=3/cpu=32/mvcc-range-keys=global was skipped due to (test_runner.go:723).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-DtmrCWsFKpuCVCTvi1NWWd5lkWo= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-09065271eb9d0144d --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-10522176-1686721617-83-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-06-14T07:40:46Z},{Key=Name,Value=teamcity-10522176-1686721617-83-n4cpu32-0003}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-10522176-1686721617-83-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-06-14T07:40:46Z},{Key=Name,Value=teamcity-10522176-1686721617-83-n4cpu32-0003}] --user-data file:///tmp/aws-startup-script2597642130 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1056476003 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 13addf52fe9b979f5cc3aeabac518058ec7715ff:

test ycsb/F/nodes=3/cpu=32 was skipped due to (test_runner.go:723).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-DtmrCWsFKpuCVCTvi1NWWd5lkWo= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-09065271eb9d0144d --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-10522176-1686721617-86-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-06-14T07:42:05Z},{Key=Name,Value=teamcity-10522176-1686721617-86-n4cpu32-0002},{Key=Roachprod,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-10522176-1686721617-86-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-06-14T07:42:05Z},{Key=Name,Value=teamcity-10522176-1686721617-86-n4cpu32-0002},{Key=Roachprod,Value=true}] --user-data file:///tmp/aws-startup-script3355906568 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1131056271 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 13addf52fe9b979f5cc3aeabac518058ec7715ff:

test ycsb/F/nodes=3/cpu=32/mvcc-range-keys=global was skipped due to (test_runner.go:723).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-DtmrCWsFKpuCVCTvi1NWWd5lkWo= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-09065271eb9d0144d --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2023-06-14T07:43:28Z},{Key=Name,Value=teamcity-10522176-1686721617-89-n4cpu32-0004},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-10522176-1686721617-89-n4cpu32},{Key=Lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2023-06-14T07:43:28Z},{Key=Name,Value=teamcity-10522176-1686721617-89-n4cpu32-0004},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-10522176-1686721617-89-n4cpu32},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script3891233728 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3120674806 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 13addf52fe9b979f5cc3aeabac518058ec7715ff:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:723).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,created=2023-06-14t17_17_35z,roachprod=true,cluster=teamcity-10522175-1686721589-153-n9cpu8-geo,lifetime=12h0m0s,arch=amd64 --metadata-from-file startup-script=/tmp/gce-startup-script145198312 --project cockroach-ephemeral --boot-disk-size=32GB --zone eu-west-1b teamcity-10522175-1686721589-153-n9cpu8-geo-0007 teamcity-10522175-1686721589-153-n9cpu8-geo-0008 teamcity-10522175-1686721589-153-n9cpu8-geo-0009]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/eu-west-1b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 3c55b81a4e64b62b7a1a59fad76c8d8a39f867a0:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:723).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,arch=amd64,created=2023-06-15t17_31_10z,roachprod=true,cluster=teamcity-10539670-1686807946-151-n9cpu8-geo,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script528562231 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west-2b teamcity-10539670-1686807946-151-n9cpu8-geo-0004 teamcity-10539670-1686807946-151-n9cpu8-geo-0005 teamcity-10539670-1686807946-151-n9cpu8-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-west-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 41a817f0d4a22f921e3d291f08394eaa8bce9adb:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,created=2023-06-16t17_21_31z,roachprod=true,cluster=teamcity-10556171-1686894353-141-n9cpu8-geo,lifetime=12h0m0s,arch=amd64 --metadata-from-file startup-script=/tmp/gce-startup-script2119394602 --project cockroach-ephemeral --boot-disk-size=32GB --zone eu-west-1b teamcity-10556171-1686894353-141-n9cpu8-geo-0007 teamcity-10556171-1686894353-141-n9cpu8-geo-0008 teamcity-10556171-1686894353-141-n9cpu8-geo-0009]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/eu-west-1b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 795dd78a4c02f04184192b61c30bcf8fd980b5ea:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-10571450-1686980790-148-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-06-17t17_19_38z --metadata-from-file startup-script=/tmp/gce-startup-script2779632296 --project cockroach-ephemeral --boot-disk-size=32GB --zone eu-west-1b teamcity-10571450-1686980790-148-n9cpu8-geo-0007 teamcity-10571450-1686980790-148-n9cpu8-geo-0008 teamcity-10571450-1686980790-148-n9cpu8-geo-0009]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/eu-west-1b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 795dd78a4c02f04184192b61c30bcf8fd980b5ea:

test import/tpcc/warehouses=4000/geo was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-standard-16 --labels usage=roachtest,arch=amd64,created=2023-06-18t16_24_35z,roachprod=true,cluster=teamcity-10578490-1687067210-128-n8cpu16-geo,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script1657298688 --project cockroach-ephemeral --boot-disk-size=32GB --zone asia-northeast1-b teamcity-10578490-1687067210-128-n8cpu16-geo-0005 teamcity-10578490-1687067210-128-n8cpu16-geo-0006]
Output: WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20230616'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'N2_CPUS' exceeded.  Limit: 32.0 in region asia-northeast1.
	metric name = compute.googleapis.com/n2_cpus
	limit name = N2-CPUS-per-project-region
	limit = 32.0
	dimensions = region: asia-northeast1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 795dd78a4c02f04184192b61c30bcf8fd980b5ea:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-10578490-1687067210-157-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-06-18t17_41_19z --metadata-from-file startup-script=/tmp/gce-startup-script1088924195 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west-2b teamcity-10578490-1687067210-157-n9cpu8-geo-0004 teamcity-10578490-1687067210-157-n9cpu8-geo-0005 teamcity-10578490-1687067210-157-n9cpu8-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-west-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 9633594f139d80d1555127a09f1ecd12dd1d9b11:

test allocbench/nodes=7/cpu=8/kv/r=0/access=skew was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-10585330-1687153522-23-n8cpu8,lifetime=12h0m0s,arch=amd64,created=2023-06-19t06_52_57z --metadata-from-file startup-script=/tmp/gce-startup-script1440746448 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-10585330-1687153522-23-n8cpu8-0001 teamcity-10585330-1687153522-23-n8cpu8-0002 teamcity-10585330-1687153522-23-n8cpu8-0003 teamcity-10585330-1687153522-23-n8cpu8-0004 teamcity-10585330-1687153522-23-n8cpu8-0005 teamcity-10585330-1687153522-23-n8cpu8-0006 teamcity-10585330-1687153522-23-n8cpu8-0007 teamcity-10585330-1687153522-23-n8cpu8-0008]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-10585330-1687153522-23-n8cpu8-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-10585330-1687153522-23-n8cpu8-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-10585330-1687153522-23-n8cpu8-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-10585330-1687153522-23-n8cpu8-0004].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-10585330-1687153522-23-n8cpu8-0006].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-10585330-1687153522-23-n8cpu8-0007].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20230616'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Internal error. Please try again or contact Google Support. (Code: '-942956615557168346')
 - Internal error. Please try again or contact Google Support. (Code: '5432646839441647037'): exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 9633594f139d80d1555127a09f1ecd12dd1d9b11:

test allocbench/nodes=7/cpu=8/kv/r=0/ops=skew was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-10585330-1687153522-28-n8cpu8,lifetime=12h0m0s,arch=amd64,created=2023-06-19t07_02_36z --metadata-from-file startup-script=/tmp/gce-startup-script4260336079 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-10585330-1687153522-28-n8cpu8-0001 teamcity-10585330-1687153522-28-n8cpu8-0002 teamcity-10585330-1687153522-28-n8cpu8-0003 teamcity-10585330-1687153522-28-n8cpu8-0004 teamcity-10585330-1687153522-28-n8cpu8-0005 teamcity-10585330-1687153522-28-n8cpu8-0006 teamcity-10585330-1687153522-28-n8cpu8-0007 teamcity-10585330-1687153522-28-n8cpu8-0008]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-10585330-1687153522-28-n8cpu8-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-10585330-1687153522-28-n8cpu8-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-10585330-1687153522-28-n8cpu8-0004].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-10585330-1687153522-28-n8cpu8-0005].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-10585330-1687153522-28-n8cpu8-0006].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-10585330-1687153522-28-n8cpu8-0007].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-10585330-1687153522-28-n8cpu8-0008].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20230616'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Internal error. Please try again or contact Google Support. (Code: '-7183523703517825313'): exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 9633594f139d80d1555127a09f1ecd12dd1d9b11:

test import/tpcc/warehouses=4000/geo was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-standard-16 --labels usage=roachtest,arch=amd64,created=2023-06-19t16_22_02z,roachprod=true,cluster=teamcity-10585330-1687153522-134-n8cpu16-geo,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script4286332067 --project cockroach-ephemeral --boot-disk-size=32GB --zone asia-northeast1-b teamcity-10585330-1687153522-134-n8cpu16-geo-0005 teamcity-10585330-1687153522-134-n8cpu16-geo-0006]
Output: WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20230616'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'N2_CPUS' exceeded.  Limit: 32.0 in region asia-northeast1.
	metric name = compute.googleapis.com/n2_cpus
	limit name = N2-CPUS-per-project-region
	limit = 32.0
	dimensions = region: asia-northeast1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 9633594f139d80d1555127a09f1ecd12dd1d9b11:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-10585330-1687153522-157-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-06-19t17_11_08z --metadata-from-file startup-script=/tmp/gce-startup-script146431593 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-10585330-1687153522-157-n9cpu8-geo-0001 teamcity-10585330-1687153522-157-n9cpu8-geo-0002 teamcity-10585330-1687153522-157-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 76da6c7dfb4d71ad614715ec5e230a6cc76fbd0e:

test kv0/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-NRxqtAz64gqmnxau_RZTyn2vA24= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-004c6ad7121a8d5a7 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2023-06-20T08:34:18Z},{Key=Name,Value=teamcity-10594791-1687239917-36-n4cpu96-0004},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-10594791-1687239917-36-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2023-06-20T08:34:18Z},{Key=Name,Value=teamcity-10594791-1687239917-36-n4cpu96-0004},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-10594791-1687239917-36-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] --user-data file:///tmp/aws-startup-script2667094872 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3487741683 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 76da6c7dfb4d71ad614715ec5e230a6cc76fbd0e:

test kv95/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-NRxqtAz64gqmnxau_RZTyn2vA24= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-004c6ad7121a8d5a7 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2023-06-20T08:35:37Z},{Key=Name,Value=teamcity-10594791-1687239917-39-n4cpu96-0003},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-10594791-1687239917-39-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2023-06-20T08:35:37Z},{Key=Name,Value=teamcity-10594791-1687239917-39-n4cpu96-0003},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-10594791-1687239917-39-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] --user-data file:///tmp/aws-startup-script267499895 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3412662961 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 76da6c7dfb4d71ad614715ec5e230a6cc76fbd0e:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,arch=amd64,created=2023-06-20t17_41_25z,roachprod=true,cluster=teamcity-10594790-1687239941-156-n9cpu8-geo,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script2472204344 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west-2b teamcity-10594790-1687239941-156-n9cpu8-geo-0004 teamcity-10594790-1687239941-156-n9cpu8-geo-0005 teamcity-10594790-1687239941-156-n9cpu8-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-west-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@srosenberg srosenberg mentioned this issue Jun 20, 2023
Merged
craig bot pushed a commit that referenced this issue Jun 21, 2023
@srosenberg
Merge #105234
0b1a45f 
105234: roachprod: add aws AZ override for c6id.24xlarge r=renatolabs a=srosenberg

Since the bump to new instance types in GCE and AWS [1], we are still experiencing occasional cluster creation issues owing to "insufficient capacity". GCE quota has already been bumped, with `asia-northeast1` being the latest, and hopefully last.

The most recent cluster creation in AWS is owing to "insufficient capacity" of `c6id.24xlarge` in us-east-2a. As a workaround, we extend the existing zone override to place `c6id.24xlarge` into us-east-2b, which
allegedly has sufficient capacity.

Note, the long-term fix is to rework how cluster creation retry currently operates, by effectively trying other AZs.

[1] #104419

Epic: none
Fixes: #78601 (comment)

Release note: None

Co-authored-by: Stan Rosenberg <[email protected]>
@craig craig bot closed this as completed in a3c62d5 Jun 21, 2023
@blathers-crl blathers-crl bot mentioned this issue Jun 21, 2023
Merged
@blathers-crl blathers-crl bot mentioned this issue Jun 21, 2023
Merged
blathers-crl bot pushed a commit that referenced this issue Jun 21, 2023
@srosenberg
roachprod: add aws AZ override for c6id.24xlarge
f0252ac 
Since the bump to new instance types in GCE and AWS [1],
we are still experiencing occasional cluster creation
issues owing to "insufficient capacity". GCE quota has
already been bumped, with `asia-northeast1` being the
latest, and hopefully last.

The most recent cluster creation in AWS is owing to
"insufficient capacity" of `c6id.24xlarge` in us-east-2a.
As a workaround, we extend the existing zone override
to place `c6id.24xlarge` into us-east-2b, which
allegedly has sufficient capacity.

Note, the long-term fix is to rework how cluster creation
retry currently operates, by effectively trying other AZs.

[1] #104419

Epic: none
Fixes: #78601 (comment)

Release note: None
srosenberg added a commit to srosenberg/cockroach that referenced this issue Jun 22, 2023
@srosenberg
roachprod: add aws AZ override for c6id.24xlarge
0bf70e0 
Since the bump to new instance types in GCE and AWS [1],
we are still experiencing occasional cluster creation
issues owing to "insufficient capacity". GCE quota has
already been bumped, with `asia-northeast1` being the
latest, and hopefully last.

The most recent cluster creation in AWS is owing to
"insufficient capacity" of `c6id.24xlarge` in us-east-2a.
As a workaround, we extend the existing zone override
to place `c6id.24xlarge` into us-east-2b, which
allegedly has sufficient capacity.

Note, the long-term fix is to rework how cluster creation
retry currently operates, by effectively trying other AZs.

[1] cockroachdb#104419

Epic: none
Fixes: cockroachdb#78601 (comment)

Release note: None
@renatolabs renatolabs mentioned this issue Apr 3, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. sync-me-8 T-testeng TestEng Team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

roachprod: add aws AZ override for c6id.24xlarge srosenberg/cockroach
2 participants
@jlinder @cockroach-teamcity