Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

server: TestStatusAPIStatements failed #69557

Closed
cockroach-teamcity opened this issue Aug 30, 2021 · 2 comments · Fixed by #69591
Closed

server: TestStatusAPIStatements failed #69557

cockroach-teamcity opened this issue Aug 30, 2021 · 2 comments · Fixed by #69591
Assignees
@xinhaoz
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

server.TestStatusAPIStatements failed with artifacts on master @ 35d2c82f8cd0d77f0487bcdbfbfd12d351293679:

                        App:                     "$ internal-update-system-locations",
                        Stats:                   roachpb.TransactionStatistics{
                            Count:      5,
                            MaxRetries: 0,
                            NumRows:    roachpb.NumericStat{Mean:1, SquaredDiffs:0},
                            ServiceLat: roachpb.NumericStat{Mean:0.0010518996, SquaredDiffs:1.6713673053199977e-08},
                            RetryLat:   roachpb.NumericStat{},
                            CommitLat:  roachpb.NumericStat{Mean:0.00031961699999999996, SquaredDiffs:2.9242306039999974e-09},
                            BytesRead:  roachpb.NumericStat{},
                            RowsRead:   roachpb.NumericStat{},
                            ExecStats:  roachpb.ExecStats{},
                        },
                        AggregatedTs: time.Time{
                            wall: 0x0,
                            ext:  63765903600,
                            loc:  (*time.Location)(nil),
                        },
                        TransactionFingerprintID: 0x78db05059f42614b,
                    },
                    NodeID: 0,
                },
                {
                    StatsData: roachpb.CollectedTransactionStatistics{
                        StatementFingerprintIDs: {0x2e180fb94d8632da},
                        App:                     "$ internal-update-system-locations",
                        Stats:                   roachpb.TransactionStatistics{
                            Count:      1,
                            MaxRetries: 0,
                            NumRows:    roachpb.NumericStat{Mean:1, SquaredDiffs:0},
                            ServiceLat: roachpb.NumericStat{Mean:0.002254526, SquaredDiffs:0},
                            RetryLat:   roachpb.NumericStat{},
                            CommitLat:  roachpb.NumericStat{Mean:2.279e-05, SquaredDiffs:0},
                            BytesRead:  roachpb.NumericStat{},
                            RowsRead:   roachpb.NumericStat{},
                            ExecStats:  roachpb.ExecStats{},
                        },
                        AggregatedTs: time.Time{
                            wall: 0x0,
                            ext:  63765903600,
                            loc:  (*time.Location)(nil),
                        },
                        TransactionFingerprintID: 0x817bb2f5cb878505,
                    },
                    NodeID: 0,
                },
            },
        }
    panic.go:613: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestStatusAPIStatements658941336
--- FAIL: TestStatusAPIStatements (7.43s)
Reproduce

To reproduce, try: 

make stressrace TESTS=TestStatusAPIStatements PKG=./pkg/server TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Parameters in this failure:

  • GOFLAGS=-parallel=4

/cc @cockroachdb/server couchand

This test on roachdash | Improve this report!
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Aug 30, 2021
@blathers-crl blathers-crl bot added the T-server-and-security DB Server & Security label Aug 30, 2021
@knz knz removed T-server-and-security DB Server & Security T-kv KV Team T-sql-observability labels Aug 30, 2021
@knz
Copy link
Contributor

knz commented Aug 30, 2021

Interestingly enough, this looks like a KV and/or SQL regression.
The error is a KV error in the "statements" SQL API.

Unsure how to route this? cc @tbg @maryliag

[09:00:45] :	 [Step 2/2] E210830 06:59:55.486971 302776 kv/kvclient/kvcoord/txn_interceptor_committer.go:479  [-] 1  making txn commit explicit failed for "sql txn" meta={id=6daaa29b pri=0.03016070 epo=0 ts=1630306795.470094412,1 min=1630306795.457691018,0 seq=15} lock=true stat=STAGING rts=1630306795.470094412,1 wto=false gul=1630306795.957691018,0 ifw=14: EndTxn hasSystemKey=true, but hasSystemConfigTrigger=false

@maryliag
Copy link
Contributor

it's a SQL issue, we will take a look at it

@xinhaoz xinhaoz mentioned this issue Aug 30, 2021
Merged
craig bot pushed a commit that referenced this issue Aug 30, 2021
@xinhaoz @Azhng
Merge #69591 #69592
e76382b 
69591: sql: deflake TestStatusAPIStatements and TestStatusAPICombinedStatements r=xinhaoz a=xinhaoz

Fixes: #69557

As mentioned in #69533, we have a race condition in tests
where we request statements stats with start=now. In these
tests we expect to see no results, but because in-memory
stats have the aggregated_ts field set on iterator return,
depending on the time the test is run we might see
results returned.

For example, suppose we insert stats at 15:45. If we then request
stats at 16:00, requesting only stats aggregated after or at the
current time, i.e. start=16:00, the aggregated_ts for in-memory
stats will be set to 16:00 and thus we will see results
returned.

To deflake these tests, we stub the aggregated_ts field to
a predetermined value.

Release justification: non-production code changes
Release note: None

69592: sql: proper version gate sql stats r=maryliag,ajwerner a=Azhng

Previously, SQL Stats's implementation for version gating is faulty.
This means that SQL Stats's job monitor would attempt to start sql
stats compaction job in an incompatible cluster.
This commit fixed the faulty implementation.

Resolves #69459
Resolves #69544
Resolves #69565

Release justification: Category 2: Bug fixes and low-risk updates to
new functionality

Release note: None

Co-authored-by: Xin Hao Zhang <[email protected]>
Co-authored-by: Azhng <[email protected]>
@craig craig bot closed this as completed in c724737 Aug 30, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@xinhaoz xinhaoz

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

sql: deflake TestStatusAPIStatements and TestStatusAPICombinedStatements xinhaoz/cockroach
4 participants
@knz @maryliag @xinhaoz @cockroach-teamcity