Could shell PME tenant: Failed to connect to MSI.

[lubaihua33]

Describe the bug

When I deleted the role assignment for the scope in PME tenant through the cloud shell, I got the following errror:
Failed to connect to MSI. Please make sure MSI is configured correctly.
Get Token request returned: <Response [400]>

My command is "az role assignment delete --assignee $identityPrincipalId --scope $instanceRGId". The scop and identity are all in PME tenant. There is no the failure when running 'az role assignment delete' in MSFT tenant.

"az role assignment create" works in both PME tenant and MSFT tenant through Cloud shell.

Related command

az role assignment delete

Errors

Failed to connect to MSI. Please make sure MSI is configured correctly.
Get Token request returned: <Response [400]>

Issue script & Debug output

az role assignment delete --assignee $identityPrincipalId --scope $instanceRGId --debug
cli.knack.cli: Command arguments: ['role', 'assignment', 'delete', '--assignee', '9edefc98-1f93-48b4-8ad0-173231f09989', '--scope', '/subscriptions/46cad376-9887-4ccc-b869-7b47bfd97b54/resourceGroups/aitl-prod3-ins-1', '--debug']
cli.knack.cli: init debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7f25657fd280>, <function OutputProducer.on_global_arguments at 0x7f25656b8c10>, <function CLIQuery.on_global_arguments at 0x7f25656581f0>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'role': ['azure.cli.command_modules.role']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name Load Time Groups Commands
cli.azure.cli.core: role 0.005 17 61
cli.azure.cli.core: Total (1) 0.005 17 61
cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_next']
cli.azure.cli.core: Loading extensions:
cli.azure.cli.core: Name Load Time Groups Commands Directory
cli.azure.cli.core: ai-examples 0.113 1 1 /usr/lib/python3.9/site-packages/azure-cli-extensions/ai-examples
cli.azure.cli.core: Total (1) 0.113 1 1
cli.azure.cli.core: Loaded 18 groups, 62 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command : role assignment delete
cli.azure.cli.core: Command table: role assignment delete
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7f2564ab5820>]
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/home/baihua/.azure/commands/2024-11-27.06-58-54.role_assignment_delete.1907.log'.
az_command_data_logger: command args: role assignment delete --assignee {} --scope {} --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument..add_subscription_parameter at 0x7f2564adf550>]
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument..add_ids_arguments at 0x7f2564a84550>, <function register_cache_arguments..add_cache_arguments at 0x7f2564aa3430>, <function register_upcoming_breaking_change_info..update_breaking_change_info at 0x7f2564aa34c0>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7f25656b8ca0>, <function CLIQuery.handle_query_parameter at 0x7f2565658280>, <function register_ids_argument..parse_ids_arguments at 0x7f2564aa33a0>]
cli.azure.cli.core.commands.client_factory: Getting management service client client_type=AuthorizationManagementClient
urllib3.connectionpool: Starting new HTTP connection (1): localhost:50342
urllib3.connectionpool: http://localhost:50342 "POST /oauth2/token HTTP/1.1" 200 2401
msrestazure.azure_active_directory: MSI: Retrieving a token from http://localhost:50342/oauth2/token, with payload {'resource': 'https://management.core.windows.net/'}
cli.azure.cli.core.util: Retrieving token for resource https://graph.microsoft.com/
urllib3.connectionpool: Starting new HTTP connection (1): localhost:50342
urllib3.connectionpool: http://localhost:50342 "POST /oauth2/token HTTP/1.1" 400 126
msrestazure.azure_active_directory: MSI: Retrieving a token from http://localhost:50342/oauth2/token, with payload {'resource': 'https://graph.microsoft.com/'}
msrestazure.azure_active_directory: MSI: Failed to retrieve a token from 'http://localhost:50342/oauth2/token' with an error of '400 Client Error: Bad Request for url: http://localhost:50342/oauth2/token'. This could be caused by the MSI extension not yet fully provisioned.
cli.azure.cli.core.auth.adal_authentication: throw requests.exceptions.HTTPError when doing MSIAuthentication:
Traceback (most recent call last):
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/auth/adal_authentication.py", line 75, in set_token
super().set_token()
File "/usr/lib64/az/lib/python3.9/site-packages/msrestazure/azure_active_directory.py", line 598, in set_token
self.scheme, _, self.token = get_msi_token(self.resource, self.port, self.msi_conf)
File "/usr/lib64/az/lib/python3.9/site-packages/msrestazure/azure_active_directory.py", line 486, in get_msi_token
result.raise_for_status()
File "/usr/lib64/az/lib/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://localhost:50342/oauth2/token

cli.azure.cli.core.azclierror: Traceback (most recent call last):
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/auth/adal_authentication.py", line 75, in set_token
super().set_token()
File "/usr/lib64/az/lib/python3.9/site-packages/msrestazure/azure_active_directory.py", line 598, in set_token
self.scheme, _, self.token = get_msi_token(self.resource, self.port, self.msi_conf)
File "/usr/lib64/az/lib/python3.9/site-packages/msrestazure/azure_active_directory.py", line 486, in get_msi_token
result.raise_for_status()
File "/usr/lib64/az/lib/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://localhost:50342/oauth2/token

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/auth/adal_authentication.py", line 87, in set_token
.format(err.response.status, err.response.reason))
AttributeError: 'Response' object has no attribute 'status'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib64/az/lib/python3.9/site-packages/knack/cli.py", line 233, in invoke
cmd_result = self.invocation.execute(args)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/init.py", line 666, in execute
raise ex
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/init.py", line 733, in _run_jobs_serially
results.append(self._run_job(expanded_arg, cmd_copy))
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/init.py", line 703, in _run_job
result = cmd_copy(params)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/init.py", line 336, in call
return self.handler(*args, **kwargs)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
return op(**command_args)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/command_modules/role/custom.py", line 539, in delete_role_assignments
assignments = _search_role_assignments(cmd.cli_ctx, assignments_client, definitions_client,
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/command_modules/role/custom.py", line 554, in _search_role_assignments
assignee_object_id = _resolve_object_id(cli_ctx, assignee, fallback_to_object_id=True)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/command_modules/role/custom.py", line 1544, in _resolve_object_id
object_id, _ = _resolve_object_id_and_type(cli_ctx, assignee, fallback_to_object_id=fallback_to_object_id)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/command_modules/role/custom.py", line 1558, in _resolve_object_id_and_type
result = list(client.service_principal_list(
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/command_modules/role/_msgrpah/_graph_client.py", line 184, in service_principal_list
result = self._send("GET", "/servicePrincipals" + _filter_to_query(filter))
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/command_modules/role/_msgrpah/_graph_client.py", line 57, in _send
r = send_raw_request(self._cli_ctx, method, url, resource=self._resource, uri_parameters=param,
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/util.py", line 988, in send_raw_request
token_info, _, _ = profile.get_raw_token(resource)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/_profile.py", line 414, in get_raw_token
msi_creds = MsiAccountTypes.msi_auth_factory(MsiAccountTypes.system_assigned, identity_id,
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/_profile.py", line 747, in msi_auth_factory
return MSIAuthenticationWrapper(resource=resource)
File "/usr/lib64/az/lib/python3.9/site-packages/msrestazure/azure_active_directory.py", line 592, in init
self.set_token()
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/auth/adal_authentication.py", line 89, in set_token
raise AzureResponseError('Failed to connect to MSI. Please make sure MSI is configured correctly.\n'
azure.cli.core.azclierror.AzureResponseError: Failed to connect to MSI. Please make sure MSI is configured correctly.
Get Token request returned: <Response [400]>

cli.azure.cli.core.azclierror: Failed to connect to MSI. Please make sure MSI is configured correctly.
Get Token request returned: <Response [400]>
az_command_data_logger: Failed to connect to MSI. Please make sure MSI is configured correctly.
Get Token request returned: <Response [400]>
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7f2564ab5a60>]
az_command_data_logger: exit code: 1
cli.main: Command ran in 40.400 seconds (init: 0.111, invoke: 40.290)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.client: Accumulated 0 events. Flush the clients.
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 4093 in cache file under /home/baihua/.azure/telemetry/20241127065934580
telemetry.main: Begin creating telemetry upload process.
telemetry.process: Creating upload process: "/usr/bin/python3.9 /usr/lib/az/lib/python3.9/site-packages/azure/cli/telemetry/init.py /home/baihua/.azure /home/baihua/.azure/telemetry/20241127065934580"
telemetry.process: Return from creating process 1945
telemetry.main: Finish creating telemetry upload process.

Expected behavior

No error and the role assignment has been deleted.

Environment Summary

az version
{
"azure-cli": "2.65.0",
"azure-cli-core": "2.65.0",
"azure-cli-telemetry": "1.1.0",
"extensions": {
"ai-examples": "0.2.5",
"ml": "2.30.1",
"ssh": "2.0.5"
}
}

Additional context

No response

[azure-client-tools-bot-prd]

Hi @lubaihua33,

2.65.0 is not the latest Azure CLI(2.67.0).

If you haven't already attempted to do so, please upgrade to the latest Azure CLI version by following https://learn.microsoft.com/en-us/cli/azure/update-azure-cli.

[yonzhan]

Thank you for opening this issue, we will look into it.

[jiasli]

I don't think it is possible that az role assignment create succeeds but az role assignment delete fails.

The debug log indicates the managed identity endpoint on Cloud Shell is not working as expected:

msrestazure.azure_active_directory: MSI: Retrieving a token from http://localhost:50342/oauth2/token, with payload {'resource': 'https://graph.microsoft.com/'}
msrestazure.azure_active_directory: MSI: Failed to retrieve a token from 'http://localhost:50342/oauth2/token' with an error of '400 Client Error: Bad Request for url: http://localhost:50342/oauth2/token'. This could be caused by the MSI extension not yet fully provisioned.

I will reach out to Cloud Shell team. If you are a Microsoft employee, please contact me internally on Teams.

[jiasli]

As discussed offline, the create command is

az role assignment create --assignee-object-id xxx --assignee-principal-type ServicePrincipal --scope xxx --role "Network Contributor"

Since both --assignee-object-id and --assignee-principal-type are provided, this command makes no Microsoft Graph query (#19219).

The delete command is

az role assignment delete --assignee xxx --scope xxx --role "Contributor"

As az role assignment delete doesn't support --assignee-object-id (#30436), a Microsoft Graph query is made. CLI tries to get an access token for Microsoft Graph https://graph.microsoft.com/ from Cloud Shell's managed identity endpoint http://localhost:50342/oauth2/token but fails.

[jiasli]

Possible workarounds or solutions:

1. Use Azure CLI locally. You may install the Azure CLI for SAW from software center.
2. Cloud Shell should fix this issue and allow getting access token for Microsoft Graph.
3. Use az rest to directly call Role Assignments - Delete REST API.
4. Azure CLI should support --assignee-object-id for az role assignment list/delete to allow bypassing Microsoft Graph query ([Feature Request] az role assignment list/delete: Support --assignee-object-id #30436).

[lubaihua33]

Possible workarounds or solutions:

1. Use Azure CLI locally. You may install the Azure CLI for SAW from software center.
2. Cloud Shell should fix this issue and allow getting access token for Microsoft Graph.
3. Use az rest to directly call Role Assignments - Delete REST API.
4. Azure CLI should support --assignee-object-id for az role assignment list/delete to allow bypassing Microsoft Graph query ([Feature Request] az role assignment list/delete: Support --assignee-object-id #30436).

Thank you @jiasli for your investigation and mitigation steps.