Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Handle missing SSH key error #69

Merged
merged 3 commits into from
Mar 11, 2021
Merged

Handle missing SSH key error #69

merged 3 commits into from
Mar 11, 2021

Conversation

andersy005
Copy link
Member

No description provided.

@mnlevy1981
Copy link
Contributor

Is there an issue ticket for the "missing ssh key error"? Is it an easy error to reproduce?

@andersy005
Copy link
Member Author

Is there an issue ticket for the "missing ssh key error"? Is it an easy error to reproduce?

I didn't open a ticket :( but this is in part one of the issues we noticed when helping Cecile. You may recall that she didn't have the SSH key pair set up. The existing try... except block was catching two exceptions only and the exception thrown when the SSH key pair is missing is different.

If you want to reproduce the error on your VM, try running the latest version of jupyter-forward on it without the SSH key pair, and let me know what you get.

@mnlevy1981
Copy link
Contributor

If you want to reproduce the error on your VM, try running the latest version of jupyter-forward on it without the SSH key pair, and let me know what you get.

This is the part that confused me when we were talking to Cecile as well - if I use jupyter-forward without the SSH key pair, it prompts me for a password :)

@mnlevy1981
Copy link
Contributor

Okay, the good news is that if I remove my authorized_keys file on a CGD machine, then I get the same error message Cecile was seeing from jupyter-forward:

$ jupyter-forward andre.cgd.ucar.edu
********************************************** Authentication **********************************************
[14:47:01] Authenticating user (******) from client (******) to remote host                   core.py:66
           (andre.cgd.ucar.edu)
           ❌ Failed to Authenticate your connection                                              core.py:87
           ❌ Failed to Authenticate your connection                                              core.py:87

The bad news is that this branch doesn't fix it:

$ jupyter-forward andre.cgd.ucar.edu
********************************************** Authentication **********************************************
[14:45:58] Authenticating user (******) from client (******) to remote host                   core.py:65
           (andre.cgd.ucar.edu)
[14:45:59] ❌ Failed to Authenticate your connection                                              core.py:83
           ❌ Failed to Authenticate your connection                                              core.py:83
           ❌ Failed to Authenticate your connection                                              core.py:83

@andersy005
Copy link
Member Author

andersy005 commented Mar 4, 2021 •
edited
Loading

@mnlevy1981, can you try again with this branch? I removed the pass and just raises the exception as a debugging process. What do you get?

@mnlevy1981
Copy link
Contributor

With 101edf5:

  File "/Users/mlevy/miniconda3/envs/jupyter-forward-dev/bin/jupyter-forward", line 33, in <module>
    sys.exit(load_entry_point('jupyter-forward', 'console_scripts', 'jupyter-forward')())
  File "/Users/mlevy/NO_BACKUP/codes/jupyter-forward/jupyter_forward/cli.py", line 97, in main
    typer.run(app())
  File "/Users/mlevy/miniconda3/envs/jupyter-forward-dev/lib/python3.9/site-packages/typer/main.py", line 214, in __call__
    return get_command(self)(*args, **kwargs)
  File "/Users/mlevy/miniconda3/envs/jupyter-forward-dev/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/mlevy/miniconda3/envs/jupyter-forward-dev/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/mlevy/miniconda3/envs/jupyter-forward-dev/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/mlevy/miniconda3/envs/jupyter-forward-dev/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/mlevy/miniconda3/envs/jupyter-forward-dev/lib/python3.9/site-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "/Users/mlevy/NO_BACKUP/codes/jupyter-forward/jupyter_forward/cli.py", line 83, in start
    runner = RemoteRunner(
  File "<string>", line 11, in __init__
  File "/Users/mlevy/NO_BACKUP/codes/jupyter-forward/jupyter_forward/core.py", line 72, in __post_init__
    raise exc
  File "/Users/mlevy/NO_BACKUP/codes/jupyter-forward/jupyter_forward/core.py", line 70, in __post_init__
    self.session.open()
  File "/Users/mlevy/miniconda3/envs/jupyter-forward-dev/lib/python3.9/site-packages/fabric/connection.py", line 636, in open
    self.client.connect(**kwargs)
  File "/Users/mlevy/miniconda3/envs/jupyter-forward-dev/lib/python3.9/site-packages/paramiko/client.py", line 435, in connect
    self._auth(
  File "/Users/mlevy/miniconda3/envs/jupyter-forward-dev/lib/python3.9/site-packages/paramiko/client.py", line 764, in _auth
    raise saved_exception
  File "/Users/mlevy/miniconda3/envs/jupyter-forward-dev/lib/python3.9/site-packages/paramiko/client.py", line 740, in _auth
    self._transport.auth_publickey(username, key)
  File "/Users/mlevy/miniconda3/envs/jupyter-forward-dev/lib/python3.9/site-packages/paramiko/transport.py", line 1580, in auth_publickey
    return self.auth_handler.wait_for_response(my_event)
  File "/Users/mlevy/miniconda3/envs/jupyter-forward-dev/lib/python3.9/site-packages/paramiko/auth_handler.py", line 250, in wait_for_response
    raise e
paramiko.ssh_exception.AuthenticationException: Authentication failed.

If I go back to pass for the first try / except block, then the exception I get is

Traceback (most recent call last):
  File "/Users/mlevy/miniconda3/envs/jupyter-forward-dev/bin/jupyter-forward", line 33, in <module>
    sys.exit(load_entry_point('jupyter-forward', 'console_scripts', 'jupyter-forward')())
  File "/Users/mlevy/NO_BACKUP/codes/jupyter-forward/jupyter_forward/cli.py", line 97, in main
    typer.run(app())
  File "/Users/mlevy/miniconda3/envs/jupyter-forward-dev/lib/python3.9/site-packages/typer/main.py", line 214, in __call__
    return get_command(self)(*args, **kwargs)
  File "/Users/mlevy/miniconda3/envs/jupyter-forward-dev/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/mlevy/miniconda3/envs/jupyter-forward-dev/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/mlevy/miniconda3/envs/jupyter-forward-dev/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/mlevy/miniconda3/envs/jupyter-forward-dev/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/mlevy/miniconda3/envs/jupyter-forward-dev/lib/python3.9/site-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "/Users/mlevy/NO_BACKUP/codes/jupyter-forward/jupyter_forward/cli.py", line 83, in start
    runner = RemoteRunner(
  File "<string>", line 11, in __init__
  File "/Users/mlevy/NO_BACKUP/codes/jupyter-forward/jupyter_forward/core.py", line 83, in __post_init__
    raise exc
  File "/Users/mlevy/NO_BACKUP/codes/jupyter-forward/jupyter_forward/core.py", line 79, in __post_init__
    loc_transport.auth_interactive_dumb(self.session.user, _authentication_handler)
  File "/Users/mlevy/miniconda3/envs/jupyter-forward-dev/lib/python3.9/site-packages/paramiko/transport.py", line 1656, in auth_interactive_dumb
    return self.auth_interactive(username, handler, submethods)
  File "/Users/mlevy/miniconda3/envs/jupyter-forward-dev/lib/python3.9/site-packages/paramiko/transport.py", line 1633, in auth_interactive
    return self.auth_handler.wait_for_response(my_event)
  File "/Users/mlevy/miniconda3/envs/jupyter-forward-dev/lib/python3.9/site-packages/paramiko/auth_handler.py", line 250, in wait_for_response
    raise e
paramiko.ssh_exception.BadAuthenticationType: Bad authentication type; allowed types: ['publickey', 'gssapi-keyex', 'gssapi-with-mic', 'password']

So I think the problem is in the loc_transport.auth_interactive_dumb(self.session.user, _authentication_handler) line. Maybe the CGD machines need auth_interactive instead of auth_interactive_dumb() for some reason?

@mnlevy1981
Add failback if auth_interactive_dumb fails
14b3391 
When loc_transport.auth_interactive_dumb() returns an exception (which was
happening on the CGD machines when a user did not set up sshkeys), fallback to
loc_transport.auth_password() instead.

I don't fully understand why auth_interactive_dumb() was failing on the CGD
machines, so I don't know if this is a general solution or if I got lucky that
it works on the CGD machines but future machines may reject both
auth_interactive_dumb() and auth_password()... hopefully it's the former.
@mnlevy1981
Copy link
Contributor

14b3391 is sufficient for me to use jupyter-forward to connect to andre even without sshkeys set up, and it still works on CISL machines, my local VMs, etc. I reverted most of the changes that were already on this branch to keep the overall difference with main as small as possible.

I'll admit that I don't really understand why it works, so I'm not confident that it is the best way to fix the problem. (I also don't really understand why the error occurs in the first place.) I've opened a ticket with the CGD help desk, hoping to figure out if the ssh server is configured differently on andre than, say, casper... knowing what is causing the problem would probably help us figure out if this is the proper fix. But, lacking that information, at least we have something that works :)

@andersy005 I'll leave this in its current state for you to look at on Monday unless the feedback I get from CGD helps identify a better solution.

andersy005
andersy005 commented Mar 7, 2021
View reviewed changes
Copy link
Member Author

@andersy005 andersy005 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your help with this, @mnlevy1981! I am going to keep this open until you receive a response from CGD folks.

@mnlevy1981
Copy link
Contributor

Thank you for your help with this, @mnlevy1981! I am going to keep this open until you receive a response from CGD folks.

I did hear back from CGD (I think the emails were from @jrya7 though I'm pretty bad at connecting email addresses and github ids). I cut out a paragraph and a parenthetical aside from his reply, but the key bits are:

I think the paramiko's auth_interactive_dumb() function fails for the CGD
machines where DUO is not required as I think this function is more for 2FA authentication:
https://www.kite.com/python/docs/paramiko.Transport.auth_interactive_dumb

I'd say for now that the best path would be to accept the PR and keep an eye
out for people who report authentication issues, making sure to gather the
machine they are trying to use and what login methods that machine uses.

So I think we're good to merge this in.

@andersy005
Copy link
Member Author

Thank you!

@andersy005 andersy005 merged commit d5c1f21 into main Mar 11, 2021
@andersy005 andersy005 deleted the fix-missing-ssh-key-error branch March 11, 2021 16:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@andersy005 @mnlevy1981