Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

upgrade/upgrades: TestUpgradeSchemaChangerElements failed #98062

Closed
cockroach-teamcity opened this issue Mar 6, 2023 · 3 comments · Fixed by #100598
Closed

upgrade/upgrades: TestUpgradeSchemaChangerElements failed #98062

cockroach-teamcity opened this issue Mar 6, 2023 · 3 comments · Fixed by #100598
Assignees
@fqazi
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Milestone
23.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Mar 6, 2023 •
edited by cockroach-jira-scripts
Loading

upgrade/upgrades.TestUpgradeSchemaChangerElements failed with artifacts on master @ 1d4feea096e9714b083cca5cda8b89706ef9e35b:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 293683 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x88
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x38
Log preceding fatal error 

=== RUN   TestUpgradeSchemaChangerElements
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/3538dab3e8bcdb43d30e6320ed019d4d/logTestUpgradeSchemaChangerElements1322792266
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestUpgradeSchemaChangerElements/running_schema_change_that_will_have_a_deprecated_waitForSchemaChangerElementMigration_element

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

Jira issue: CRDB-25059
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Mar 6, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone Mar 6, 2023
@blathers-crl blathers-crl bot added the T-sql-schema-deprecated Use T-sql-foundations instead label Mar 6, 2023
@fqazi fqazi self-assigned this Mar 6, 2023
@cockroach-teamcity
Copy link
Member Author

upgrade/upgrades.TestUpgradeSchemaChangerElements failed with artifacts on master @ 1b162d1b274eec7b307fbbfca7294460bfdef025:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 338193 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x88
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x38
Log preceding fatal error 

=== RUN   TestUpgradeSchemaChangerElements
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/9a9dbc5a7e380eaf5ef4ce24e3fe45dd/logTestUpgradeSchemaChangerElements3200980430
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestUpgradeSchemaChangerElements/running_schema_change_that_will_have_a_deprecated_waitForSchemaChangerElementMigration_element

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

smg260 pushed a commit to smg260/cockroach that referenced this issue Mar 14, 2023
upgrade/upgrades: skip TestUpgradeSchemaChangerElements
b03f9f1 
Refs: cockroachdb#98062

Reason: flaky test

Generated by bin/skip-test.

Release justification: non-production code changes

Release note: None
@smg260 smg260 mentioned this issue Mar 14, 2023
Merged
craig bot pushed a commit that referenced this issue Mar 14, 2023
@gtr @matthewtodd
@adityamaru @msbutler
Merge #97138 #98410 #98464 #98510 #98579
421d136 
97138: ui: add error code to stmt and txn insights details pages r=gtr a=gtr

Part of: #87785.

Previously, the stmt and txn insights details pages did not show any
further information for failed executions. This commit adds an "error
code" column to the insights table for a failed execution in the stmt
and txn insights details pages. Additionally, a "status" column was
added to the stmt and txn workload insights tables which is either
"Completed" or "Failed".

Future work involves adding the error message string in addition to the
error code but it needs to be redacted first. Additionally, the txn
status is missing the implementation of a "Cancelled" status.

Note to reviewers: only consider the second commit, as the first is 
required to get the txn status.

- Loom [demo](https://www.loom.com/share/e82b97ff9f034d82b98640170eb54408).

Release note (ui change): Adds error code column to the insights table
for a failed execution in the stmt and txn insights details page. Adds
status column to the stmt and txn workload insights tables.

98410: cluster-ui: tenants use sqlstats-supplied regions r=matthewtodd a=matthewtodd

Fixes #98056.

As of #95449, the SQL Activity pages in the DB Console can draw regions information directly from the sqlstats tables, rather than having to translate node IDs to regions on page load.
    
Here, we make that switch, but for non-system tenants only, because:
    
1. The ephemeral nature of serverless nodes made this view-time mapping especially problematic in that context. (See further notes in #95449.)
    
2. The system-tenant views also include KV node IDs in a special Regions/Nodes column, which we are unable to recreate given the backend storage structure. (Future design work might suggest removing these node IDs altogether, for a unified UI.)

# Screenshots!
## Statements, with and without regions filter
<img width="1372" alt="statements" src="https://user-images.githubusercontent.com/5261/225033247-739df90a-9173-4aab-a666-a61a1ceeb579.png">
<img width="1372" alt="statements - filtered" src="https://user-images.githubusercontent.com/5261/225033271-1c0d0f82-3dd4-48ea-bdef-11f19af97a85.png">

## Statement details
<img width="1372" alt="statement details" src="https://user-images.githubusercontent.com/5261/225033338-6dff4a6e-a4a3-48c6-863a-84f1375b0a61.png">

## Transactions, with and without regions filter
<img width="1372" alt="transactions" src="https://user-images.githubusercontent.com/5261/225033366-65f44e95-3549-47cc-b0f2-67ad48a1a1fa.png">
<img width="1372" alt="transactions - filtered" src="https://user-images.githubusercontent.com/5261/225033391-50b9a2dc-e9a1-457b-84b1-837426eba35e.png">

## Transaction details
<img width="1372" alt="transaction details" src="https://user-images.githubusercontent.com/5261/225033505-3fdeceef-35dc-4e06-af25-ab4d0c53518f.png">

Release note: None

98464: jobs,upgrades: add migration to backfill job_info table r=dt a=adityamaru

This change adds a migration and corresponding cluster version
after which every job entry in the system.jobs table will have its
Payload and Progress written to two rows in the system.job_info table.

Informs: #97762

Release note: None

98510: backupccl: update restore/nodeshutdown tests to use new roachtest framework r=adityamaru a=msbutler

The restore/nodeshutdown tests have been using a very old workload that will not be restorable when #93804 lands. This patch changes the restore/nodeshutdown workload to a 80GB tpce restore and moves the tests to run on aws instead of gcp.

Release note: None

Epic: None

98579: upgrade/upgrades: skip TestUpgradeSchemaChangerElements r=smg260 a=smg260

Refs: #98062

Reason: flaky test

Generated by bin/skip-test.

Release justification: non-production code changes

Release note: None
Epic: None

Co-authored-by: gtr <[email protected]>
Co-authored-by: Matthew Todd <[email protected]>
Co-authored-by: adityamaru <[email protected]>
Co-authored-by: Michael Butler <[email protected]>
Co-authored-by: Miral Gadani <[email protected]>
@rafiss
Copy link
Collaborator

rafiss commented Mar 23, 2023

theory from Faizan: this might be a test design bug, we probably are hitting transaction retries

@rafiss rafiss removed the GA-blocker label Mar 27, 2023
@fqazi fqazi mentioned this issue Apr 7, 2023
Merged
@fqazi
Copy link
Collaborator

fqazi commented Apr 7, 2023

Fixed via #100598

@fqazi fqazi closed this as completed Apr 7, 2023
@exalate-issue-sync exalate-issue-sync bot added T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) and removed T-sql-schema-deprecated Use T-sql-foundations instead labels May 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fqazi fqazi

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
None yet
Milestone
23.1
Development

Successfully merging a pull request may close this issue.

upgrade/upgrades: fix TestUpgradeSchemaChangerElements race condition fqazi/cockroach
3 participants
@rafiss @fqazi @cockroach-teamcity