Public Key Authentication Issue

[bigjew92]

Please answer the following

If it is a terminal issue then please go through wiki
https://github.com/PowerShell/Win32-OpenSSH/wiki/TTY-PTY-support-in-Windows-OpenSSH

"OpenSSH for Windows" version
((Get-Item (Get-Command sshd).Source).VersionInfo.FileVersion)
--7.7.2.0

Server OperatingSystem
((Get-ItemProperty "HKLM:\SOFTWARE\Microsoft\Windows nt\CurrentVersion\" -Name ProductName).ProductName)
--Windows 8.1 Enterprise

Client OperatingSystem
--CentOS 7

What is failing
--Public Key Authenticatino

Expected output
--Client able to log in via SSH

Actual output
--Permission denied. Able to authenticate via password

[bagajjal]

Follow the instructions here https://github.com/PowerShell/Win32-OpenSSH/wiki/Troubleshooting-Steps

[bigjew92]

Thank you. The logs are showing it is getting my publickey, but still not letting me in. I'm now getting a write failed: broken pipe error when trying to log in. I have my permissions on the .ssh folder and authorized_keys file set properly with just my account and System having access.

[bagajjal]

Please share ssh client (.\ssh.exe -vvv user@ip) and sshd logs (DEBUG3).

[bigjew92]

...
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /root/.ssh/id_rsa (0x7f2d71420740),
debug2: key: /root/.ssh/id_dsa ((nil)),
debug2: key: /root/.ssh/id_ecdsa ((nil)),
debug2: key: /root/.ssh/id_ed25519 ((nil)),
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug3: start over, passed a different list publickey,password,keyboard-interactive
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 535
debug2: input_userauth_pk_ok: fp cc:6c:52:e4:aa:68:cd:6e:00:fb:af:47:e3:01:e2:1f
debug3: sign_and_send_pubkey: RSA cc:6c:52:e4:aa:68:cd:6e:00:fb:af:47:e3:01:e2:1f
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey).
Authenticated to 10.32.192.43 ([10.32.192.43]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug1: Requesting [email protected]
debug1: Entering interactive session.
Write failed: Broken pipe
$

It accepts the key, but when trying to open a session, fails.

[bagajjal]

Try this,

1. stop the sshd service (net stop sshd)
2. Start the sshd in debug mode (./sshd.exe -ddd).
   We should see the error message why it's closing the session.

Are you able to login using password based authentication? Ideally it should also fail.

[bigjew92]

Sorry for the late reply, have not been able to work on this. When running in debug mode, I am not able to get in using my public key OR my password. If I am running as a service, I am able to log in only by using my password, not key.

C:\Program Files\OpenSSH-Win64>sshd.exe -ddd > C:\Users\username\Desktop\output.txt
debug2: load_server_config: filename PROGRAMDATA\ssh/sshd_config
debug2: load_server_config: done config len = 152
debug2: parse_server_config: config PROGRAMDATA\ssh/sshd_config len 152
debug3: PROGRAMDATA\ssh/sshd_config:38 setting AuthorizedKeysFile .ssh/authorized_keys
debug3: PROGRAMDATA\ssh/sshd_config:76 setting Subsystem sftp sftp-server.exe
debug1: sshd version OpenSSH_for_Windows_7.7, LibreSSL 2.6.4
debug1: private host key #0: ssh-rsa SHA256:86iQP6LQyXad7wWuJtH8hy7C5XJXMcpMVrhHMqg0m0w
debug1: private host key #1: ecdsa-sha2-nistp256 SHA256:bA5mOnxDW831TD19qpBnQdV4gS0BpBLUZlAskdpc1jQ
debug1: private host key #2: ssh-ed25519 SHA256:eQjpwjdoMtyX1NPpa+tbL4S6NnvdBn5R3pBpAvJ6Dz8
debug1: rexec_argv[0]='sshd.exe'
debug1: rexec_argv[1]='-ddd'
debug2: fd 3 setting O_NONBLOCK
debug3: sock_set_v6only: set socket 3 IPV6_V6ONLY
debug1: Bind to port 22 on ::.
Server listening on :: port 22.
debug2: fd 4 setting O_NONBLOCK
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug3: fd 5 is not O_NONBLOCK
debug1: Server will not fork when running in debugging mode.
debug3: send_rexec_state: entering fd = 8 config len 152
debug3: ssh_msg_send: type 0
debug3: send_rexec_state: done
Connection from 10.32.192.234 port 46912 on 10.32.192.43 port 22
debug1: Client protocol version 2.0; client software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug1: Local version string SSH-2.0-OpenSSH_for_Windows_7.7
debug2: fd 5 setting O_NONBLOCK
debug3: spawning "C:\Program Files\OpenSSH-Win64\sshd.exe" "-ddd" "-y"
debug2: Network child is on pid 3804
debug3: send_rexec_state: entering fd = 4 config len 152
debug3: ssh_msg_send: type 0
debug3: send_rexec_state: done
debug3: ssh_msg_send: type 0
debug3: ssh_msg_send: type 0
debug3: preauth child monitor started
debug3: recv_rexec_state: entering fd = 3
debug3: ssh_msg_recv entering
debug3: recv_rexec_state: done
debug2: parse_server_config: config PROGRAMDATA\ssh/sshd_config len 152
debug3: PROGRAMDATA\ssh/sshd_config:38 setting AuthorizedKeysFile .ssh/authorized_keys
debug3: PROGRAMDATA\ssh/sshd_config:76 setting Subsystem sftp sftp-server.exe
debug1: sshd version OpenSSH_for_Windows_7.7, LibreSSL 2.6.4
debug3: ssh_msg_recv entering
debug3: ssh_msg_recv entering
debug2: fd 5 setting O_NONBLOCK
debug1: list_hostkey_types: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug3: send packet: type 20 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug3: receive packet: type 20 [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug2: local server KEXINIT proposal [preauth]
debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellma
n-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1 [preauth]
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected] [preauth]
debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected] [preauth]
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],umac
[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],umac
[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
debug2: compression ctos: none [preauth]
debug2: compression stoc: none [preauth]
debug2: languages ctos: [preauth]
debug2: languages stoc: [preauth]
debug2: first_kex_follows 0 [preauth]
debug2: reserved 0 [preauth]
debug2: peer client KEXINIT proposal [preauth]
debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellma
n-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,ext-info-c
[preauth]
debug2: host key algorithms: [email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-
sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],[email protected],ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss
[preauth]
debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],aes128-cbc,aes192-cbc,aes256-cbc [preauth]
debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],aes128-cbc,aes192-cbc,aes256-cbc [preauth]
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],umac
[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],umac
[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
debug2: compression ctos: none,[email protected],zlib [preauth]
debug2: compression stoc: none,[email protected],zlib [preauth]
debug2: languages ctos: [preauth]
debug2: languages stoc: [preauth]
debug2: first_kex_follows 0 [preauth]
debug2: reserved 0 [preauth]
debug1: kex: algorithm: curve25519-sha256 [preauth]
debug1: kex: host key algorithm: ecdsa-sha2-nistp256 [preauth]
debug1: kex: client->server cipher: [email protected] MAC: compression: none [preauth]
debug1: kex: server->client cipher: [email protected] MAC: compression: none [preauth]
debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
debug3: receive packet: type 30 [preauth]
debug3: mm_key_sign entering [preauth]
debug3: mm_request_send entering: type 6 [preauth]
debug3: mm_key_sign: waiting for MONITOR_ANS_SIGN [preauth]
debug3: mm_request_receive_expect entering: type 7 [preauth]
debug3: mm_request_receive entering [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 6
debug3: mm_answer_sign
debug3: mm_answer_sign: hostkey proof signature 0000003AEFDC30A0(100)
debug3: mm_request_send entering: type 7
debug2: monitor_read: 6 used once, disabling now
debug3: send packet: type 31 [preauth]
debug3: send packet: type 21 [preauth]
debug2: set_newkeys: mode 1 [preauth]
debug1: rekey after 134217728 blocks [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
debug3: send packet: type 7 [preauth]
debug3: receive packet: type 21 [preauth]
debug1: SSH2_MSG_NEWKEYS received [preauth]
debug2: set_newkeys: mode 0 [preauth]
debug1: rekey after 134217728 blocks [preauth]
debug1: KEX done [preauth]
debug3: receive packet: type 5 [preauth]
debug3: send packet: type 6 [preauth]
debug3: receive packet: type 50 [preauth]
debug1: userauth-request for user DOMAIN\\username service ssh-connection method none [preauth]
debug1: attempt 0 failures 0 [preauth]
debug3: mm_getpwnamallow entering [preauth]
debug3: mm_request_send entering: type 8 [preauth]
debug3: mm_getpwnamallow: waiting for MONITOR_ANS_PWNAM [preauth]
debug3: mm_request_receive_expect entering: type 9 [preauth]
debug3: mm_request_receive entering [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 8
debug3: mm_answer_pwnamallow
debug2: parse_server_config: config reprocess config len 152
debug3: mm_answer_pwnamallow: sending MONITOR_ANS_PWNAM: 1
debug3: mm_request_send entering: type 9
debug2: monitor_read: 8 used once, disabling now
debug2: input_userauth_request: setting up authctxt for DOMAIN\\username [preauth]
debug3: mm_inform_authserv entering [preauth]
debug3: mm_request_send entering: type 4 [preauth]
debug2: input_userauth_request: try method none [preauth]
debug3: userauth_finish: failure partial=0 next methods="publickey,password,keyboard-interactive" [preauth]
debug3: send packet: type 51 [preauth]
debug3: receive packet: type 50 [preauth]
debug1: userauth-request for user DOMAIN\\username service ssh-connection method publickey [preauth]
debug1: attempt 1 failures 0 [preauth]
debug2: input_userauth_request: try method publickey [preauth]
debug1: userauth_pubkey: test pkalg rsa-sha2-512 pkblob RSA SHA256:fABsQ1viTOPYR3TaDa/96xyBO0qkN+hKKsMQ/GAW6sA [preauth]
debug3: mm_key_allowed entering [preauth]
debug3: mm_request_send entering: type 22 [preauth]
debug3: mm_key_allowed: waiting for MONITOR_ANS_KEYALLOWED [preauth]
debug3: mm_request_receive_expect entering: type 23 [preauth]
debug3: mm_request_receive entering [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 4
debug3: mm_answer_authserv: service=ssh-connection, style=
debug2: monitor_read: 4 used once, disabling now
debug3: mm_request_receive entering
debug3: monitor_read: checking request 22
debug3: mm_answer_keyallowed entering
debug3: mm_answer_keyallowed: key_from_blob: 0000003AEFDC30A0
debug1: trying public key file C:\Users\username\.ssh/authorized_keys
debug3: mm_answer_keyallowed: publickey authentication test: RSA key is not allowed
Failed publickey for DOMAIN\username from 10.32.192.234 port 46912 ssh2: RSA SHA256:fABsQ1viTOPYR3TaDa/96xyBO0qkN+hKKsMQ/GAW6sA
debug3: mm_request_send entering: type 23
debug2: userauth_pubkey: authenticated 0 pkalg rsa-sha2-512 [preauth]
debug3: userauth_finish: failure partial=0 next methods="publickey,password,keyboard-interactive" [preauth]
debug3: send packet: type 51 [preauth]
debug3: receive packet: type 50 [preauth]
debug1: userauth-request for user DOMAIN\\username service ssh-connection method keyboard-interactive [preauth]
debug1: attempt 2 failures 1 [preauth]
debug2: input_userauth_request: try method keyboard-interactive [preauth]
debug1: keyboard-interactive devs [preauth]
debug1: auth2_challenge: user=DOMAIN\\username devs= [preauth]
debug1: kbdint_alloc: devices '' [preauth]
debug2: auth2_challenge_start: devices [preauth]
debug3: userauth_finish: failure partial=0 next methods="publickey,password,keyboard-interactive" [preauth]
debug3: send packet: type 51 [preauth]
debug3: receive packet: type 50 [preauth]
debug1: userauth-request for user DOMAIN\\username service ssh-connection method password [preauth]
debug1: attempt 3 failures 2 [preauth]
debug2: input_userauth_request: try method password [preauth]
debug3: mm_auth_password entering [preauth]
debug3: mm_request_send entering: type 12 [preauth]
debug3: mm_auth_password: waiting for MONITOR_ANS_AUTHPASSWORD [preauth]
debug3: mm_request_receive_expect entering: type 13 [preauth]
debug3: mm_request_receive entering [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 12
debug3: lookup_principal_name: Successfully discovered explicit principal name: 'domain\username'=>'EMAIL'
debug1: Windows authentication failed for user: EMAIL domain: (null) error: 1326
debug3: mm_answer_authpassword: sending result 0
debug3: mm_request_send entering: type 13
Failed password for DOMAIN\username from 10.32.192.234 port 46912 ssh2
debug3: mm_auth_password: user not authenticated [preauth]
debug3: userauth_finish: failure partial=0 next methods="publickey,password,keyboard-interactive" [preauth]
debug3: send packet: type 51 [preauth]
Connection closed by authenticating user DOMAIN\\username 10.32.192.234 port 46912 [preauth]
debug1: do_cleanup [preauth]
debug1: monitor_read_log: child log fd closed
debug3: mm_request_receive entering
debug1: do_cleanup
debug1: Killing privsep child 3804

C:\Program Files\OpenSSH-Win64>

[bagajjal]

I am guessing it's the problem with your authorized_keys file.

debug1: trying public key file C:\Users\username.ssh/authorized_keys
debug3: mm_answer_keyallowed: publickey authentication test: RSA key is not allowed
Failed publickey for DOMAIN\username from 10.32.192.234 port 46912 ssh2: RSA

Verify that your authorized_keys entries looks like this,
ssh-rsa <key> <username>

Also you can try generating new keys and using them.

[bigjew92]

Well, I realized I was using the wrong key, changed to the correct key and it worked in interactive more, but still not working while running as a service, getting the same error as reported previously.

[bagajjal]

Then it should be an issue with the ACLs.
To fix ACLs, try these instructions (https://github.com/PowerShell/Win32-OpenSSH/wiki/OpenSSH-utility-scripts-to-fix-file-permissions#fixuserfilepermissionsps1).

If it didn't solve then follow these steps,

1. Stop sshd (net stop sshd)

2. Enable file based logging. Add below lines to "%programdata%\ssh\sshd_config"
   SyslogFacility LOCAL0
   LogLevel DEBUG3

3. Start the sshd (net start sshd)

4. Establish ssh connection (.\ssh.exe -vvv user@ip)

5. Now look for the error log in "%programdata%\ssh\logs\sshd.log"

[bigjew92]

Followed directions in the link, thank you. Everything came back successful, still running into the same error.

as for the other troubleshooting fix you mentioned, I am trying to ssh to this box, so I don't believe step 4 would work. I enabled those two items in the config and still nothing, nothing should up in the logs folder either.

[mgkuhn]

@bagajjal Thanks for the instructions on how to activate logging. Could you please add them for future reference to this wiki page?

For me this revealed the cause as

debug3: Bad permissions. Try removing permissions for user: S-1-5-80-3847866527-469524349-687026318-516638107-1125189541 on file C: \\Users\\mgk25\\.ssh/authorized_keys.

(Displaying the username rather than only the SID would be nicer.)

And the permissions were in my case:

C:\>icacls \Users\mgk25\.ssh\authorized_keys        
\Users\mgk25\.ssh\authorized_keys NT SERVICE\sshd:(R)                           
                                  NT AUTHORITY\SYSTEM:(I)(F)                    
                                  BUILTIN\Administrators:(I)(F)                 
                                  cheltenham\mgk25:(I)(F)                       

So that's just sshd, myself and two inherited standard SYSTEM and Administrators SIDs. Does that really have to count as "bad permissions"?

I eventually repaired it with

C:\> powershell
PS> cd 'C:\Program Files\OpenSSH-Win64'
PS> Set-ExecutionPolicy Unrestricted -Scope CurrentUser
PS> Import-Module .\OpenSSHUtils.psd1 -Force
PS> Repair-AuthorizedKeyPermission -FilePath C:\Users\mgk25\.ssh\authorized_keys

resulting in

PS> icacls \Users\mgk25\.ssh\authorized_keys
\Users\mgk25\.ssh\authorized_keys NT AUTHORITY\SYSTEM:(F)
                                  cheltenham\mgk25:(F)

[iainnicol]

@bagajjal In a sense, there is a real bug here in Repair-AuthorizedKeyPermission; this is not just user misconfiguration. The code change required was committed to this repo long ago. However, the updated code still needs to be deployed to the PowerShell Gallery.

Like @mgkuhn, my problem was the NT SERVICE\sshd:(R) permission on authorized_keys. But how did that permission get there? It was calling Repair-AuthorizedKeyPermission that added that invalid permission for me!

Indeed, originally that was deliberate. Repair-AuthorizedKeyPermission would add read access for NT SERVICE\sshd. Then a year ago, @manojampalam added privilege separation. You can see they simultaneously also removed the code which added said read access.

Now, official Windows documentation says to install OpenSSHUtils from the PowerShell Gallery, and then use the module to set correct permissions. That seems generally reasonable. Unfortunately, the most recent release of OpenSSHUtils on the PowerShell Gallery predates @manojampalam's privilege separation change. That means Repair-AuthorizedKeyPermission adds read access for NT SERVICE\ssh, which is no longer required, and moreover no longer allowed.

In conclusion, please release a new version of OpenSSHUtils to the PowerShell Gallery. Thanks.

[manojampalam]

@iainnicol thanks for catching this. @bingbing8, @maertendMSFT please follow up.
With privilege separation enabled, permissions on Windows are more aligned with those in Unix and it shouldn't take more than a couple of icacls calls to mend file permissions. Wonder if this utility is still required.

[maertendMSFT]

Please try the latest release.

[jeffwilcox]

@maertendMSFT the latest release of what? Windows, OpenSSH Server, ... ?

My team's been trying to use this as part of working on Codespaces and it's pretty hard to SSH to Windows with all of the out-of-date docs... feel free to ping me internally, to, if need be. Happy to try to help and make this better.

[mgkuhn]

The github release of OpenSSH for Windows is very far (nearly two years) ahead of the old v7.7 one that comes bundled with the Windows 10 distribution, which has caused regular confusion on github issues. Hopefully the one in Windows 20H1 will finally catch up in a couple of weeks time. Regarding the specific issue here: OpenSSH sshd rejects your authorized_keys file if it thinks the NTFS access-control list allows access to other parties, although the detailed test applied is unfortunately not well documented for the Windows port (and NTFS access control is very different from the POSIX mode bits used on other platforms where OpenSSH runs). Therefore my general advice (without knowing your particulars) is: learn how NTFS ACLs work and then check whether you can remove any unnecessary access-control entries (ACEs) from authorized_keys, and you should be fine. The underlying problem may be that very few Windows users understand NTFS ACLs and ACEs concepts, and therefore struggle to read and set them. NTFS ACLs are very useful and well worth understanding, not just for OpenSSH.

[jeffwilcox]

Got it, thanks for writing back - will see about using the project from GitHub instead as a starting point and learn about NTFS ACLs. icacls time!