Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New index backfill schema change resulted in error in newOrder: ERROR: no inbound stream connection (SQLSTATE XX000) #34738

Closed
awoods187 opened this issue Feb 8, 2019 · 4 comments
Labels
A-schema-changes C-investigation Further steps needed to qualify. C-label will change. S-3-ux-surprise Issue leaves users wondering whether CRDB is behaving properly. Likely to hurt reputation/adoption.

Comments

@awoods187
Copy link
Contributor

awoods187 commented Feb 8, 2019

Describe the problem

While running tpc-c 10k, I created a schema change. The schema change caused an error that stopped QPS.

Error: error in newOrder: ERROR: no inbound stream connection (SQLSTATE XX000)
Error:  exit status 1

image
image

To Reproduce
Setup TPC-C 1K like normal.

Set flag for new schema change implementation:
Roachprod sql $CLUSTER:1
SET CLUSTER SETTING schemachanger.bulk_index_backfill.enabled = true

Run for 6 hours.
roachprod run $CLUSTER:4 "./workload run tpcc --ramp=5m --warehouses=2000 --duration=6h --split --scatter {pgurl:1-3}"

After five minutes, in another shell, change the zone config ttl to 10 minutes
Roachprod sql $CLUSTER:1
ALTER RANGE default CONFIGURE ZONE USING gc.ttlseconds = 600;

Then create an index or non-unique on any column:
use tpcc
CREATE INDEX ON item (i_price)

Expected behavior
Index < 10 minutes
Look at the front page on admin ui metrics SQL QPS and make sure it never went to less than 80% pre index

Environment:
v2.2.0-alpha.20181217-1144-ge3e4f96

@awoods187
Copy link
Contributor Author

@awoods187
Copy link
Contributor Author

Retried this again on the same cluster after dropping the index and restarting tpc-c. I waited a bit longer but the index took much longer (I assume because of new data) and did not hose the cluster. It did reduce performance:

image
image

@vivekmenezes
Copy link
Contributor

this is related to #31361

We should close it unless we can reproduce it.

@awoods187 awoods187 added S-3-ux-surprise Issue leaves users wondering whether CRDB is behaving properly. Likely to hurt reputation/adoption. A-schema-changes C-investigation Further steps needed to qualify. C-label will change. labels Mar 5, 2019
@jordanlewis
Copy link
Member

This is 2 releases old at this point; closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-schema-changes C-investigation Further steps needed to qualify. C-label will change. S-3-ux-surprise Issue leaves users wondering whether CRDB is behaving properly. Likely to hurt reputation/adoption.
Projects
None yet
Development

No branches or pull requests

3 participants