kv/kvserver: TestTenantRateLimiter failed

[cockroach-teamcity]

kv/kvserver.TestTenantRateLimiter failed with artifacts on master @ bfdcfa00836232b910cb0fd109ebedf8913b1466:

=== RUN   TestTenantRateLimiter
    test_log_scope.go:79: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestTenantRateLimiter069762263
    test_log_scope.go:80: use -show-logs to present logs inline
    client_tenant_test.go:248: 
        	Error Trace:	client_tenant_test.go:248
        	Error:      	
        	Test:       	TestTenantRateLimiter
    panic.go:613: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestTenantRateLimiter069762263
--- FAIL: TestTenantRateLimiter (23.34s)

Reproduce

To reproduce, try:

make stressrace TESTS=TestTenantRateLimiter PKG=./pkg/kv/kvserver TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Parameters in this failure:

• GOFLAGS=-json

/cc @cockroachdb/kv ajwerner

This test on roachdash | Improve this report!

[cockroach-teamcity]

kv/kvserver.TestTenantRateLimiter failed with artifacts on master @ 85b1e92455eb2795043d71e5766b1b56dab308f2:

=== RUN   TestTenantRateLimiter
    test_log_scope.go:79: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestTenantRateLimiter750218330
    test_log_scope.go:80: use -show-logs to present logs inline
    client_tenant_test.go:248: 
        	Error Trace:	client_tenant_test.go:248
        	Error:      	
        	Test:       	TestTenantRateLimiter
    panic.go:613: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestTenantRateLimiter750218330
--- FAIL: TestTenantRateLimiter (31.25s)

Reproduce

To reproduce, try:

make stressrace TESTS=TestTenantRateLimiter PKG=./pkg/kv/kvserver TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Parameters in this failure:

• GOFLAGS=-json

/cc @cockroachdb/kv ajwerner

This test on roachdash | Improve this report!

[tbg]

The failure is on this line:

cockroach/pkg/kv/kvserver/client_tenant_test.go

Line 248 in 85b1e92

require.Contains(t, getMetrics(), makeMetricStr(int64(tooManyWrites)))

It's curious that it doesn't output anything. There should be a message "X does not contain Y", and the objects we're passing to Contains are both strings (i.e. there's no internal panic trying to do something with whatever we're passing). Also, getMetrics does not call t.Helper() so if somehow the error is from there it should get reported from there as well.

Think we'll need to repro this to see what's going on.

[stevendanna]

Also, getMetrics does not call t.Helper() so if somehow the error is from there it should get reported from there as well.

Interesting, I was also looking into this and hadn't excluded getMetrics() from what I was looking at. It seems to happen in CI under bazel a bit more often than the non-bazel job.

[cockroach-teamcity]

kv/kvserver.TestTenantRateLimiter failed with artifacts on master @ a0d6d46d3283be18483e3d449adc7d90a02649c1:

=== RUN   TestTenantRateLimiter
    test_log_scope.go:79: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestTenantRateLimiter129364174
    test_log_scope.go:80: use -show-logs to present logs inline
    client_tenant_test.go:248: 
        	Error Trace:	client_tenant_test.go:248
        	Error:      	
        	Test:       	TestTenantRateLimiter
    panic.go:613: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestTenantRateLimiter129364174
--- FAIL: TestTenantRateLimiter (31.99s)

Reproduce

To reproduce, try:

make stressrace TESTS=TestTenantRateLimiter PKG=./pkg/kv/kvserver TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Parameters in this failure:

• GOFLAGS=-json

/cc @cockroachdb/kv ajwerner

This test on roachdash | Improve this report!

[RaduBerinde]

I can repro locally. Instead of

kv_tenant_rate_limit_write_requests_admitted{store="1",tenant_id="10"} 24002

we have

kv_tenant_rate_limit_write_requests_admitted{store="1",tenant_id="10"} 24003

Some background write has snuck in and messed with out strict check. Almost surely triggered by my recent change to increase the limits.

[RaduBerinde]

It's curious that it doesn't output anything. There should be a message "X does not contain Y",

I'm not sure exactly how it happens, but I think it's because the metrics string is very long.