Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sql/tests: crdb_internal.reset_sql_stats() timeout #69541

Closed
cockroach-teamcity opened this issue Aug 29, 2021 · 6 comments
Closed

sql/tests: crdb_internal.reset_sql_stats() timeout #69541

cockroach-teamcity opened this issue Aug 29, 2021 · 6 comments
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Aug 29, 2021 •
edited by cockroach-jira-scripts
Loading

sql/tests.TestRandomSyntaxFunctions failed with artifacts on master @ 6700cf65e82a029a1ff76a75250c709aa9637170:

Random syntax error:

    rsg_test.go:776: Crash detected: server panic: statement exec timeout

Query:

        SELECT crdb_internal.reset_sql_stats() ;
Reproduce

To reproduce, try: 

make stressrace TESTS=TestRandomSyntaxFunctions PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Same failure on other branches

Internal log 

mjibson marked as alumn{us/a}; resolving to rafiss instead

/cc @cockroachdb/sql-experience rafiss

This test on roachdash | Improve this report!

Jira issue: CRDB-9647
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Aug 29, 2021
@Azhng Azhng self-assigned this Aug 29, 2021
@cockroach-teamcity
Copy link
Member Author

sql/tests.TestRandomSyntaxFunctions failed with artifacts on master @ 0b57dc40deda1206d9a1c215ffdb219bbf182a39:

Random syntax error:

    rsg_test.go:776: Crash detected: server panic: statement exec timeout

Query:

        SELECT crdb_internal.reset_sql_stats() ;
Reproduce

To reproduce, try: 

make stressrace TESTS=TestRandomSyntaxFunctions PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Same failure on other branches

Internal log 

mjibson marked as alumn{us/a}; resolving to rafiss instead

/cc @cockroachdb/sql-experience rafiss

This test on roachdash | Improve this report!

@Azhng Azhng mentioned this issue Aug 30, 2021
Merged
Azhng added a commit to Azhng/cockroach that referenced this issue Aug 30, 2021
@Azhng
sql: proper version gate sql stats
e20f0af 
Previously, SQL Stats's implementation for version gating is faulty.
This means that SQL Stats's job monitor would attempt to start sql
stats compaction job in an incompatible cluster.
This commit fixed the faulty implementation.

Resolves cockroachdb#69459
Resolves cockroachdb#69541
Resolves cockroachdb#69544
Resolves cockroachdb#69565

Release justification: Category 2: Bug fixes and low-risk updates to
new functionality

Release note: None
Azhng added a commit to Azhng/cockroach that referenced this issue Aug 30, 2021
@Azhng
sql: proper version gate sql stats
412f804 
Previously, SQL Stats's implementation for version gating is faulty.
This means that SQL Stats's job monitor would attempt to start sql
stats compaction job in an incompatible cluster.
This commit fixed the faulty implementation.

Resolves cockroachdb#69459
Resolves cockroachdb#69541
Resolves cockroachdb#69544
Resolves cockroachdb#69565

Release justification: Category 2: Bug fixes and low-risk updates to
new functionality

Release note: None
Azhng added a commit to Azhng/cockroach that referenced this issue Aug 30, 2021
@Azhng
sql: proper version gate sql stats
4c5c0aa 
Previously, SQL Stats's implementation for version gating is faulty.
This means that SQL Stats's job monitor would attempt to start sql
stats compaction job in an incompatible cluster.
This commit fixed the faulty implementation.

Resolves cockroachdb#69459
Resolves cockroachdb#69541
Resolves cockroachdb#69544
Resolves cockroachdb#69565

Release justification: Category 2: Bug fixes and low-risk updates to
new functionality

Release note: None
Azhng added a commit to Azhng/cockroach that referenced this issue Aug 30, 2021
@Azhng
sql: proper version gate sql stats
8c3bbd6 
Previously, SQL Stats's implementation for version gating is faulty.
This means that SQL Stats's job monitor would attempt to start sql
stats compaction job in an incompatible cluster.
This commit fixed the faulty implementation.

Resolves cockroachdb#69459
Resolves cockroachdb#69541
Resolves cockroachdb#69544
Resolves cockroachdb#69565

Release justification: Category 2: Bug fixes and low-risk updates to
new functionality

Release note: None
@Azhng
Copy link
Contributor

Azhng commented Aug 30, 2021

Hmm this is very odd. We are causing the SQL Engine to OOM.

        Stack trace:
        timeout: "SELECT crdb_internal.reset_sql_stats() ". currently executing: map[SELECT crdb_internal.reset_sql_stats() :8]
E210829 05:26:01.161042 1983726 sql/distsql/server.go:393  [n1,client=127.0.0.1:51006,hostssl,user=root,intExec=check-existing-sql-stats-schedule] 45  error setting up flow: root: memory budget exceeded: 10240 bytes requested, 134215680 currently allocated, 134217728 bytes in budget
    rsg_test.go:184: goroutine 1244677 [running]:
        github.com/cockroachdb/cockroach/pkg/sql/tests_test.(*verifyFormatDB).execWithTimeout(0xc0109873e0, 0xc001b19200, 0x6a36788, 0xc00356fa80, 0xc01b930570, 0x27, 0xdf8475800, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/tests/rsg_test.go:183 +0x285
        github.com/cockroachdb/cockroach/pkg/sql/tests_test.(*verifyFormatDB).exec(...)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/tests/rsg_test.go:125
        github.com/cockroachdb/cockroach/pkg/sql/tests_test.TestRandomSyntaxFunctions.func2(0x6a36788, 0xc00356fa80, 0xc0109873e0, 0xc01a88bf40, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/tests/rsg_test.go:360 +0x4da
        github.com/cockroachdb/cockroach/pkg/sql/tests_test.testRandomSyntax.func3(0x6a36788, 0xc00356fa80, 0x0, 0x133a533, 0x6a367c0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/tests/rsg_test.go:766 +0xc7
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GroupWorkers.func1(0x6a36788, 0xc00356fa80, 0xc01dc32180, 0x5a23798)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:175 +0x42
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1(0x0, 0x5a23840)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:166 +0x3a
        golang.org/x/sync/errgroup.(*Group).Go.func1(0xc01d117500, 0xc01d117530)
        	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x59
        created by golang.org/x/sync/errgroup.(*Group).Go
        	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66

@Azhng
Copy link
Contributor

Azhng commented Aug 30, 2021

This doesn't directly fails the SELECT crdb_internal.reset_sql_stats(), the OOM comes from a separate query. But still results in the execution timeout.

@Azhng
Copy link
Contributor

Azhng commented Aug 30, 2021

Here is another test timeout where OOM didn't happen:

        Stack trace:
        timeout: "SELECT crdb_internal.reset_sql_stats() ". currently executing: map[SELECT avg((-7865654628768669800):::INT8) :1 SELECT crdb_internal.reset_sql_stats() :7]
    rsg_test.go:184: goroutine 1244682 [running]:
        github.com/cockroachdb/cockroach/pkg/sql/tests_test.(*verifyFormatDB).execWithTimeout(0xc0109873e0, 0xc001b19200, 0x6a36788, 0xc00356fa80, 0xc0077a1bf0, 0x27, 0xdf8475800, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/tests/rsg_test.go:183 +0x285
        github.com/cockroachdb/cockroach/pkg/sql/tests_test.(*verifyFormatDB).exec(...)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/tests/rsg_test.go:125
        github.com/cockroachdb/cockroach/pkg/sql/tests_test.TestRandomSyntaxFunctions.func2(0x6a36788, 0xc00356fa80, 0xc0109873e0, 0xc01a88bf40, 0x699bc00, 0xc0193c6e40)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/tests/rsg_test.go:360 +0x4da
        github.com/cockroachdb/cockroach/pkg/sql/tests_test.testRandomSyntax.func3(0x6a36788, 0xc00356fa80, 0x5, 0x133a533, 0x6a367c0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/tests/rsg_test.go:766 +0xc7
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GroupWorkers.func1(0x6a36788, 0xc00356fa80, 0xc01dc32180, 0x5a23798)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:175 +0x42
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1(0x0, 0x5a23840)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:166 +0x3a
        golang.org/x/sync/errgroup.(*Group).Go.func1(0xc01d117500, 0xc01d117650)
        	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x59
        created by golang.org/x/sync/errgroup.(*Group).Go
        	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66

It's is possible that these two are separate issues .

@craig craig bot closed this as completed in 3ea128d Aug 30, 2021
@Azhng Azhng reopened this Aug 30, 2021
@cockroach-teamcity
Copy link
Member Author

sql/tests.TestRandomSyntaxFunctions failed with artifacts on master @ c1ef81f5f435b3cc5bdf8b218532e0779f03a6bf:

Random syntax error:

    rsg_test.go:776: Crash detected: server panic: statement exec timeout

Query:

        SELECT crdb_internal.reset_sql_stats() ;
Reproduce

To reproduce, try: 

make stressrace TESTS=TestRandomSyntaxFunctions PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Same failure on other branches

Internal log 

mjibson marked as alumn{us/a}; resolving to rafiss instead

/cc @cockroachdb/sql-experience rafiss

This test on roachdash | Improve this report!

@Azhng Azhng changed the title sql/tests: TestRandomSyntaxFunctions failed sql/tests: crdb_internal.reset_sql_stats() timeout Aug 31, 2021
@Azhng
Copy link
Contributor

Azhng commented Aug 31, 2021

Neither stressrace the TestRandomSyntaxFunctions nor TestPersistedSQLStatsReset was able to reproduce this failure.

@Azhng Azhng mentioned this issue Sep 2, 2021
Closed
@Azhng Azhng mentioned this issue Sep 2, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@maryliag @Azhng @cockroach-teamcity