Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SSH session may be broken or closed during the operation #730

Open
AstroProfundis opened this issue Aug 29, 2020 · 0 comments
Open

SSH session may be broken or closed during the operation #730

AstroProfundis opened this issue Aug 29, 2020 · 0 comments
Labels
category/stability Categorizes issue or PR as a stability enhancement. status/TODO Categorizes issue as we will do it. type/bug Categorizes issue as related to a bug.

Comments

@AstroProfundis
Copy link
Contributor

Bug Report

Please answer these questions before submitting your issue. Thanks!

  1. What did you do?

We store SSH sessions for hosts in context and reuse them for every SSH operations, but it is possible for the session to be closed or somehow broken before we finish all commands.

There is a cased that when ControlMaster and ControlPath is set in the SSH config of console server, random timeout error might be observed during operations like start for the cluster, while filtering only some roles / nodes avoids that error.

It's possible that when there are more instances than some number, the SSH session is used too late after it was created, and at that time it is already closed or stalled.

  1. What did you expect to see?
    We should find a better way handling SSH sessions, maybe send keepalive packets?

  2. What did you see instead?
    Some operations got random timeout error on large cluster.

  3. What version of TiUP are you using (tiup --version)?
    v1.0.8, and should be valid on v1.1.0 too.

@AstroProfundis AstroProfundis added type/bug Categorizes issue as related to a bug. category/stability Categorizes issue or PR as a stability enhancement. labels Aug 29, 2020
@lucklove lucklove added status/TODO Categorizes issue as we will do it. and removed status/WIP labels Aug 31, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category/stability Categorizes issue or PR as a stability enhancement. status/TODO Categorizes issue as we will do it. type/bug Categorizes issue as related to a bug.
Projects
None yet
Development

No branches or pull requests

3 participants