BadHostKeyException when attempting az acs kubernetes get-credentials

[jungho]

Is this a request for help?:
Yes

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)
Kubernetes 1.8.1
Canadacentral

az --version
azure-cli (2.0.31)
python -V
Python 2.7.14
python3 -V
Python 3.6.5

What happened:

Successfully created cluster, when trying to get the kubeconfig credentials, the parimiko library raises a BadHostKeyException.

Created cluster with:

az group create -n acs-cluster -l canadacentral
az acs create -n acs-cluster -g acs-cluster -t Kubernetes --master-count 1 --agent-count 2 --orchestrator-version 1.8.1 --generate-ssh-keys

Get credentials with:

az acs kubernetes get-credentials -n acs-cluster -g acs-cluster --debug

I get the following output (modified to remove subscription info)

msrest.pipeline : Configuring request: timeout=100, verify=True, cert=None
msrest.pipeline : Configuring redirects: allow=True, max=30
msrest.pipeline : Configuring proxies: ''
msrest.pipeline : Evaluate proxies against ENV settings: True
msrest.pipeline : Configuring retry: max_retries=4, backoff_factor=0.8, max_backoff=90
urllib3.connectionpool : Starting new HTTPS connection (1): management.azure.com
urllib3.connectionpool : https://management.azure.com:443 "GET /subscriptions/xxx/resourceGroups/acs-cluster/providers/Microsoft.ContainerService/containerServices/acs-cluster
?api-version=2017-07-01 HTTP/1.1" 200 None
msrest.http_logger : Request URL: 'https://management.azure.com/subscriptions/REMOVED/resourceGroups/acs-cluster/providers/Microsoft.ContainerService/containerServices/acs-cluster
?api-version=2017-07-01'
msrest.http_logger : Request method: 'GET'
msrest.http_logger : Request headers:
msrest.http_logger : 'Connection': 'keep-alive'
msrest.http_logger : 'Accept-Encoding': 'gzip, deflate'
msrest.http_logger : 'Accept': 'application/json'
msrest.http_logger : 'User-Agent': 'python/2.7.14 (Linux-4.15.14-300.fc27.x86_64-x86_64-with-fedora-27-Twenty_Seven) requests/2.18.4 msrest/0.4.27 msrest_azure/0.4.25 azure-mgmt-containerservice/3.0.1 Azu
re-SDK-For-Python AZURECLI/2.0.31'
msrest.http_logger : 'Authorization': '*****'
msrest.http_logger : 'x-ms-client-request-id': 'dfdafds'
msrest.http_logger : 'CommandName': 'acs kubernetes get-credentials'
msrest.http_logger : 'Content-Type': 'application/json; charset=utf-8'
msrest.http_logger : 'accept-language': 'en-US'
msrest.http_logger : Request body:
msrest.http_logger : None
msrest.http_logger : Response status: 200
msrest.http_logger : Response headers:
msrest.http_logger : 'Cache-Control': 'no-cache'
msrest.http_logger : 'Pragma': 'no-cache'
msrest.http_logger : 'Transfer-Encoding': 'chunked'
msrest.http_logger : 'Content-Type': 'application/json'
msrest.http_logger : 'Content-Encoding': 'gzip'
msrest.http_logger : 'Expires': '-1'
msrest.http_logger : 'Vary': 'Accept-Encoding'
msrest.http_logger : 'x-ms-correlation-request-id': '2aa92282-6bd7-4763-973d-1a4dd60582f7'
msrest.http_logger : 'x-ms-request-id': '16259863-8bac-40a4-960e-de768ff482f3'
msrest.http_logger : 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
msrest.http_logger : 'Server': 'nginx'
msrest.http_logger : 'x-ms-ratelimit-remaining-subscription-reads': '14992'
msrest.http_logger : 'x-ms-routing-request-id': 'CANADACENTRAL:20180415T122044Z:2aa92282-6bd7-4763-973d-1a4dd60582f7'
msrest.http_logger : 'X-Content-Type-Options': 'nosniff'
msrest.http_logger : 'Date': 'Sun, 15 Apr 2018 12:20:44 GMT'
msrest.http_logger : Response content:
msrest.http_logger : {
"id": "/subscriptions/REMOVED/resourcegroups/acs-cluster/providers/Microsoft.ContainerService/containerServices/acs-cluster",
"location": "canadacentral",
"name": "acs-cluster",
"type": "Microsoft.ContainerService/ContainerServices",
"properties": {
"provisioningState": "Succeeded",
"orchestratorProfile": {
"orchestratorType": "Kubernetes",
"orchestratorVersion": "1.8.1"
},
"masterProfile": {
"count": 1,
"dnsPrefix": "acs-cluste-acs-cluster-17ef54mgmt",
"vmSize": "Standard_D2_v2",
"firstConsecutiveStaticIP": "10.240.255.5",
"storageProfile": "ManagedDisks",
"fqdn": "acs-cluste-acs-cluster-17ef54mgmt.canadacentral.cloudapp.azure.com"
},
"agentPoolProfiles": [
{
"name": "agentpool0",
"count": 2,
"vmSize": "Standard_D2_v2",
"dnsPrefix": "",
"fqdn": "",
"storageProfile": "StorageAccount",
"osType": "Linux"
}
],
"linuxProfile": {
"adminUsername": "azureuser",
"ssh": {
"publicKeys": [
{
"keyData": "ssh-rsa key REMOVED"
}
]
}
},
"servicePrincipalProfile": {
"clientId": "REMOVED"
}
}
}
paramiko.transport : starting thread (client mode): 0x886dd3d0L
paramiko.transport : Local version/idstring: SSH-2.0-paramiko_2.4.1
paramiko.transport : Remote version/idstring: SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.4
paramiko.transport : Connected (version 2.0, client OpenSSH_7.2p2)
paramiko.transport : kex algos:[u'[email protected]', u'ecdh-sha2-nistp256', u'ecdh-sha2-nistp384', u'ecdh-sha2-nistp521', u'diffie-hellman-group-exchange-sha256', u'diffie-hellman-group14-sha1'] s
erver key:[u'ssh-rsa', u'rsa-sha2-512', u'rsa-sha2-256', u'ecdsa-sha2-nistp256', u'ssh-ed25519'] client encrypt:[u'[email protected]', u'aes128-ctr', u'aes192-ctr', u'aes256-ctr', u'aes128-gcm@ope
nssh.com', u'[email protected]'] server encrypt:[u'[email protected]', u'aes128-ctr', u'aes192-ctr', u'aes256-ctr', u'[email protected]', u'[email protected]'] client mac:[u'umac-64
[email protected]', u'[email protected]', u'[email protected]', u'[email protected]', u'[email protected]', u'[email protected]', u'[email protected]', u'hmac-sha2-2
56', u'hmac-sha2-512', u'hmac-sha1'] server mac:[u'[email protected]', u'[email protected]', u'[email protected]', u'[email protected]', u'[email protected]', u'um
[email protected]', u'[email protected]', u'hmac-sha2-256', u'hmac-sha2-512', u'hmac-sha1'] client compress:[u'none', u'[email protected]'] server compress:[u'none', u'[email protected]'] client lang:[u''] s
erver lang:[u''] kex follows?False
paramiko.transport : Kex agreed: ecdh-sha2-nistp256
paramiko.transport : HostKey agreed: ecdsa-sha2-nistp256
paramiko.transport : Cipher agreed: aes128-ctr
paramiko.transport : MAC agreed: hmac-sha2-256
paramiko.transport : Compression agreed: none
paramiko.transport : kex engine KexNistp256 specified hash_algo
paramiko.transport : Switch to new keys ...
('acs-cluste-acs-cluster-17ef54mgmt.canadacentral.cloudapp.azure.com', <paramiko.ecdsakey.ECDSAKey object at 0x7f99886d5c90>, <paramiko.ecdsakey.ECDSAKey object at 0x7f998b6a3850>)
Traceback (most recent call last):
File "/usr/lib64/az/lib/python2.7/site-packages/knack/cli.py", line 197, in invoke
cmd_result = self.invocation.execute(args)
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/commands/init.py", line 347, in execute
six.reraise(*sys.exc_info())
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/commands/init.py", line 319, in execute
result = cmd(params)
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/commands/init.py", line 180, in call
return super(AzCliCommand, self).call(*args, **kwargs)
File "/usr/lib64/az/lib/python2.7/site-packages/knack/commands.py", line 109, in call
return self.handler(*args, **kwargs)
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/init.py", line 420, in default_command_handler
result = op(**command_args)
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/command_modules/acs/custom.py", line 926, in k8s_get_credentials
_k8s_get_credentials_internal(name, acs_info, path, ssh_key_file)
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/command_modules/acs/custom.py", line 947, in _k8s_get_credentials_internal
'.kube/config', path_candidate, key_filename=ssh_key_file)
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/command_modules/acs/acs_client.py", line 72, in secure_copy
ssh.connect(host, username=user, pkey=pkey, sock=proxy)
File "/usr/lib64/az/lib/python2.7/site-packages/paramiko/client.py", line 409, in connect
raise BadHostKeyException(hostname, server_key, our_key)
BadHostKeyException: ('acs-cluste-acs-cluster-17ef54mgmt.canadacentral.cloudapp.azure.com', <paramiko.ecdsakey.ECDSAKey object at 0x7f99886d5c90>, <paramiko.ecdsakey.ECDSAKey object at 0x7f998b6a3850>)
paramiko.transport : EOF in transport thread

What you expected to happen:

To get credentials to access the cluster using kubectl.

How to reproduce it (as minimally and precisely as possible):

Follow the commands above.

Anything else we need to know:

I have seen this issue before, sometimes, when I downgraded the az client, it seemed to correct the issue. This time, it did not. I have also tried different regions (canadaeast) and in the past, it corrected the issue, this time it did not. I have also tried creating the cluster by specifying my own key then getting the credentials with --ssh-key-value option. Same result.

[jungho]

Note, this issue does not exist in Canada East region. It does exist in Canada Central.