Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VReplication panic on broken connection to external MySQL #5752

Closed
flopex opened this issue Jan 23, 2020 · 0 comments · Fixed by #5753
Closed

VReplication panic on broken connection to external MySQL #5752

flopex opened this issue Jan 23, 2020 · 0 comments · Fixed by #5753

Comments

@flopex
Copy link

flopex commented Jan 23, 2020

Overview of the Issue

vttablet crashes after many hours of running and a broken connection occurs between the vttablet and the external MySQL db host. It's common to see broken connections every once in a while but the crashes only seem to happen after the tablet has been left running for many hours.

Seems to be related to #5413 but was dismissed due to outdated vitess image.

Reproduction Steps

N/A

Binary version

vttablet --version
Version: 10083991b (Git branch 'master') built on Thu Jan 16 21:25:10 UTC 2020 by root@577cbc709acd using go1.12.14 linux/amd64

Operating system and Environment details

OS, Architecture, and any other information you can provide
about the environment.

  • Operating system (output of cat /etc/os-release):
  • Kernel version (output of uname -sr):
  • Architecture (output of uname -m):

Log Fragments

panic trace
https://gist.github.com/flopex/05d58c97d7066db0a474fdb97fbfc930

sougou added a commit to planetscale/vitess that referenced this issue Jan 23, 2020
Fixes vitessio#5752

DBClient was setting the connection to nil after close, which was
causing vreplication to panic because it defers a Rollback on all
connections. The safer thing is to close the connection, but not
set it to nil.

There's no test for DBClient, but I've manually verified that this
won't panic any more.

Signed-off-by: Sugu Sougoumarane <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant