[BUG][Text Analytics]Text Analytics Cilent fail to authenticate with token credential in USGOV and China

[Luyunmt]

We are running live Tests against other clouds like US Gov and Azure China Cloud. The goal is to check whether new azure sdk package work with other clouds or not.
In https://github.com/Azure/azure-sdk-for-net/blob/master/sdk/textanalytics/Azure.AI.TextAnalytics/src/TextAnalyticsClient.cs#L68 as follow.

The scope is hard code as ' https://cognitiveservices.azure.com/.default' leading to only work well in the public cloud.

The value of the scope in different clouds is follow:
Azure public cloud: https://cognitiveservices.azure.com/.default
USGOV cloud: https://cognitiveservices.azure.us/.default
China cloud: https://cognitiveservices.azure.cn/.default

AZURE_AUTHORITY_HOST setting:
USGOV : https://login.microsoftonline.us/
China: https://login.microsoftonline.cn/
@jongio @danieljurek @benbp

[maririos]

We have the same problem in FR #17192

@tg-msft do you know if there is a guidance for .NET on how to do this? or how to surface this to our customers?
For sdk tests, I see how some services have the default scope as an env variable that gets configured with the CI, but wonder how the user will know that in case they are in one of those clouds.

[maririos]

FYI @suhas92

[jongio]

@Luyunmt - Can you please give examples of how this is achieved in the sdk with other cloud endpoints?

[tg-msft]

Storage hardcodes that today so this a better question for @schaabs. If we're ready to add a common story for this, I think we should maybe stick this on the base ClientOptions or at least have a common TokenAuthenticationOptions type we can expose from any service's *ClientOptions that allow TokenCredential.

[Luyunmt]

@jongio Example in this link https://github.com/Azure/azure-sdk-for-net/blob/master/sdk/textanalytics/Azure.AI.TextAnalytics/tests/DetectLanguageTests.cs#L46 can repro this issue

[schaabs]

The client generally should handle which scopes are required to properly authorize the call. Some services (such as storage) use the same scope in all clouds which is why they are able to hard code the scope they use. Other services need to use different scopes in different clouds. In these cases, the client should, if possible, handle this. There are a couple of strategies a client can choose to determine the required scope.

First, if the service returns the required scopes or resource string via an authentication challenge (WWW-Authenticate header) the client should utilize this to determine the scope. Key Vault is an example of this and currently has it's own authentication policy which implements this discovery.

If the no such data is available through an authentication challenge, a service client may be able to determine the targeted cloud and required scope based off the resource endpoint the user specifies. However, this can break down if the service supports custom domain links, or if end users use private links.

We've avoided adding any client configuration for this up to this point as it introduces quite a bit of complexity to the user. The TokenAuthenticationOptions class @tg-msft suggests is very interesting. But we'd need to work out details such as what this means for services which don't support AAD (a shrinking minority. Also, while most azure services only use a singe scope to authenticate all calls, it's possible that different methods might require different scopes, so how would these options apply in that case.

[jongio]

@maririos - Can you see if cognitive returns the scope/resource string in an Auth challenge?

If so, does this work for default domains and resources with custom subdomain? To use AAD auth with Cognitive services, then the resource needs a custom subdomain, see: https://docs.microsoft.com/en-us/azure/cognitive-services/authentication?tabs=powershell#authenticate-with-azure-active-directory. For other services, a key is required and we therefore shouldn't have to worry about scope in other clouds.

@tg-msft - Are you comfortable with the following while @johanste fleshes out cloud environment design?

1. Ask user to set scope in TokenCredentialContext
2. Update TA and FR clients to honor the scope if set.

@johanste - Where are we with a design for cloud environment?

[tg-msft]

@schaabs - I'd think we'd only put TokenAuthenticationOptions on derived classes like SearchClientOptions if:

• they supported AAD,
• varied scopes across clouds, and
• didn't provide an API to discover the right scope.

It'd unblock things today and ideally we'd [EditorBrowsable(Never)] them over time as they added automagic discovery. We could try guessing defaults based on endpoints as well with this being an override for situations where we get it wrong.

@jongio - I'm in favor of enabling something for forward progress here. I think @schaabs will have the best idea how to do that with the least long term debt and I'll get behind his plan.

[jongio]

@maririos - How do you recommend we proceed with this?

[maririos]

We can def try the approach Ted suggested of adding the parameter to TextAnalyticsClientOptions class.
I currently don't have the bandwidth to do this, so if no one else could do it and it is a priority let me know and I'll ask for funding

[jongio]

@v-xuto - Can you please do a PR for this? @maririos can be advisor. Thanks

[v-xuto]

@jongio - Working on this.

[zedy-wj]

@maririos some of the TA features are not enabled in UsGov. Healthcare tests are failing. Should we remove UsGov cloud or do a PR for this issue?

[maririos]

Better to remove it. This is a service constrain so unless there is a way to disable tests per clouds, not much we can do

[zedy-wj]

@jongio All tests about Healthcare failed. Because feature not support in others clouds except Public. @maririos suggested that we remove other clouds. Therefore, adding scopes in UsGov and China are meaningless now. Should we hold this issue until Healthcare service support in other regions?

[maririos]

Even though the test won't pass, it will be good to make the proper changes in the TA library so users that don't need Healthcare can target other clouds

[zedy-wj]

OK, I will do a PR for this issue.

[jongio]

We are holding on fixing this until the ACR design is complete. #21603 (comment)

[zedy-wj]

Hi, @benbp , @maririos!
We have updated the code, this issue has been fixed with sovereign cloud test PR.

But there is a test named RecognizeHealthcareEntitiesBatchWithCancellation was not stable.

The pipeline run result is at here. Could you help to see this problem, any thoughts of it?

[maririos]

There is a known issue with that test => #24052

We haven't had the time to look into it in order to fix it though

[zedy-wj]

@nisha-bhatia , @maririos - Do you have any progress or plans to fix this issue?

[nisha-bhatia]

This bug fix will go out in the next release.