-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
sql/tests: TestRandomSyntaxFunctions failed #69731
Comments
@Azhng know anything about this? |
Seems like duplicated for #69541. Spent some time investigating earlier this week, had difficult time reproducing it. Any thoughts on what I should be trying next ? |
fyi the reason it duplicated is because the issue title was changed. if the title were |
there's a pretty huge goroutine dump in https://teamcity.cockroachdb.com/repository/download/Cockroach_Nightlies_RandomSyntaxTests/3397108:id/raw.0.json.txt.tgz!/artifacts/full_output.txt -- were you able to find anything helpful in there? |
this part seems related, but i can't tell if this indicates a problem or not
i'd start by making sure you are using the internal executor correctly in that part of the code |
Hm it's doing a TRUNCATE - i guess it's possible that there'd be a lot of contention on these stats tables during this test, so maybe the TRUNCATE is being retried endlessly? |
The code that uses internal executor is here Effectively, it issues two queries: TRUNCATE system.statement_statistics;
TRUNCATE system.transaction_statistics; IIUC, these two queries would create two schema change jobs that drops the indexes and cleanup afterwards. |
Ah yes, the flusher is also part of the stack trace. So it seems like it is possible that the stats flush is preventing the table from being dropped.
|
Hmm let me write a script to prove my hypothesis. |
Hmm seems like the flush is not that's blocking the table drop. I had the following script spamming unique statement fingerprints and called #!/usr/bin/env bash
UNTIL=$1
START=1
set -eu
str='SELECT 1'
echo "Spamming stmt fingerprints"
for (( i = $START; i <= $UNTIL; i++)); do
echo -n -e "$i/$UNTIL\r"
str="$str,1"
$HOME/src/github.com/cockroach/cockroach sql --insecure -e "$str" > /dev/null
done;
echo ""
echo "Done."
|
The only other piece of code that touches those two tables are SQLStatsCompactor. However, it's not present in the stack trace at all :/ |
sql/tests.TestRandomSyntaxFunctions failed with artifacts on master @ 42e5f9492d0d8d93638241303bca984fe78baae3: Random syntax error:
Query:
ReproduceTo reproduce, try: make stressrace TESTS=TestRandomSyntaxFunctions PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 Same failure on other branches
Internal log
/cc @cockroachdb/sql-experience rafiss |
sql/tests.TestRandomSyntaxFunctions failed with artifacts on master @ 44c95054cfe0108df872d2cda7952fc92259426b: Random syntax error:
Query:
ReproduceTo reproduce, try: make stressrace TESTS=TestRandomSyntaxFunctions PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 Same failure on other branches
Internal log
/cc @cockroachdb/sql-experience rafiss |
sql/tests.TestRandomSyntaxFunctions failed with artifacts on master @ e369d86b3bac2b9f40d760c3e6e49f55b7e15abe: Random syntax error:
Query:
ReproduceTo reproduce, try: make stressrace TESTS=TestRandomSyntaxFunctions PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 Same failure on other branches
Internal log
/cc @cockroachdb/sql-experience rafiss |
sql/tests.TestRandomSyntaxFunctions failed with artifacts on master @ 642e44afbe6098f25022618be76c6f7b6b97df45: Random syntax error:
Query:
ReproduceTo reproduce, try: make stressrace TESTS=TestRandomSyntaxFunctions PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 Same failure on other branches
Internal log
/cc @cockroachdb/sql-experience rafiss |
sql/tests.TestRandomSyntaxFunctions failed with artifacts on master @ 2a7ef521a38d923cca9ec96e229237bbedc0c5fa: Random syntax error:
Query:
ReproduceTo reproduce, try: make stressrace TESTS=TestRandomSyntaxFunctions PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 Same failure on other branches
Internal log
/cc @cockroachdb/sql-experience rafiss |
@Azhng This is failing pretty reliably, so I think we'll need to do something about it. If you are confident there isn't a major bug with the function, then can you update the randomized test so that it doesn't try using this function? |
Previously, crdb_internal.reset_sql_stats() causes timeout in TestRandomSyntaxFunctions. This is very unlikely due to implementation of the function, and it is likely caused by contentions. This commit skip the tests for crdb_internal.reset_sql_stats() to prevent nightly failures. Related cockroachdb#69731 Release justification: Non-production code changes Release note: None
69806: kv/kvserver: use generalized engine keys in debug printing r=AlexTalks a=AlexTalks Previously the CLI debug command only printed `MVCCKey`s, resulting in the debug printer to error on key decoding whenever a `LockTableKey` was encountered. By switching to the more generalized `EngineKey` (and utilizing the existing MVCC key formatting whenever the key has an MVCC version), we can increase visibility into our debug logs while investigating issues. Related to #69414 Release justification: Non-production bug fix Release note: None 69944: stats: add a histogram version number r=rytaft a=rytaft This commit adds a histogram version number to the `HistogramData` proto. This will allow us to identify what logic was used to construct a particular histogram and possibly debug future issues. Release note: None Release justification: Low risk, high benefit change to existing functionality. 69963: sql: skip reset sql stats in TestRandomSyntaxFunctions r=maryliag,rafiss a=Azhng Previously, crdb_internal.reset_sql_stats() causes timeout in TestRandomSyntaxFunctions. This is very unlikely due to implementation of the function, and it is likely caused by contentions. This commit skip the tests for crdb_internal.reset_sql_stats() to prevent nightly failures. Related #69731 Release justification: Non-production code changes Release note: None 69967: vendor: bump Pebble to 6c12d67b83e6 r=jbowens a=jbowens ``` 6c12d67 internal/metamorphic: randomize FormatMajorVersion e82fb10 db: randomize format major version in unit tests 535b8d6 db: add FormatUpgrade event to EventListener 53dda0f db: introduce format major version 8ec1a49 vfs/atomicfs: add ReadMarker daf93f0 sstable: Free cache value when checksum type is corrupt d89613d metamorphic: randomly use disk for tests e3b6bec metamorphic: transform percentage of SINGLEDEL ops to DELETE ops 41239f8 db: add test demonstrating current SINGLEDEL behavior ``` Release note: none Release justification: non-production code changes, and bug fix necessary for a release blocker. 69974: backupccl: set sqlstats testing knobs for scheduled job test r=maryliag,miretskiy a=Azhng Previsouly, backupccl tests did not set sql stats AOST testing knob to override the AOST behavior. This causes sql stats error stack trace to show up in backupccl tests. This commit added sql stats testing knobs for backupccl test helpers to mitigate this. Release justification: Non-production code changes Release note: None Co-authored-by: Alex Sarkesian <[email protected]> Co-authored-by: Rebecca Taft <[email protected]> Co-authored-by: Azhng <[email protected]> Co-authored-by: Jackson Owens <[email protected]>
sql/tests.TestRandomSyntaxFunctions failed with artifacts on master @ 62c5be51bbb3beeb51638032871365585d69f75d: Random syntax error:
Query:
ReproduceTo reproduce, try: make stressrace TESTS=TestRandomSyntaxFunctions PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 Same failure on other branches
Internal log
/cc @cockroachdb/sql-experience rafiss |
Previously, crdb_internal.reset_sql_stats() causes timeout in TestRandomSyntaxFunctions. This is very unlikely due to implementation of the function, and it is likely caused by contentions. This commit skip the tests for crdb_internal.reset_sql_stats() to prevent nightly failures. Related cockroachdb#69731 Release justification: Non-production code changes Release note: None
sql/tests.TestRandomSyntaxFunctions failed with artifacts on master @ 63679e7bf4a0244c6e46381751543cfe3f32388c:
Random syntax error:
Query:
Reproduce
To reproduce, try:
Same failure on other branches
Internal log
/cc @cockroachdb/sql-experience rafiss
This test on roachdash | Improve this report!
The text was updated successfully, but these errors were encountered: