known-issue: ExpiredAuthenticationToken when running a long-running operation
#20867
Comments
|
@op7ic From the error code, it seems the token is expired and somehow not refreshed. Would you please provide the debug log so that we can further investigate? |
|
hi @magodo that's correct - it exactly what happens mid-way through build. I'll get the log and upload it here. |
|
@op7ic I believe the root cause (the auth token not renewing correctly in all cases) has been fixed in v3.47.0, would you mind updating to the latest version and seeing if that works for you? |
|
@tombuildsstuff sure let me try. In the meantime here is the last error from debug log: |
|
@tombuildsstuff Probabaly it is v3.48.0, per #20894? |
|
@magodo I believe 3.48.0 is not yet published but 3.47.0 is as per https://registry.terraform.io/providers/hashicorp/azurerm/latest. I'll try 3.47 first and report back to see if there is an improvement. |
|
@tombuildsstuff / @magodo I can confirm that 3.47.0 is still the same. I'll test 3.48 once it arrives but looking at commitment code for the release it seems that it should work ;) |
|
Sounds good - since #20894 has been merged and this'll go out in this weeks release - I'm going to close this issue for the moment and assign this to this weeks milestone, once the release is out then a comment will be posted - but if this still an issue in v3.48 then please let us know and we can take another look 👍 |
|
This functionality has been released in v3.48.0 of the Terraform Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you! |
|
@tombuildsstuff |
|
@manicminer mind taking a look into this one? |
|
@garbology Would you be able to provide a debug log showing the error and requests that led to it? Also, can you confirm your system clock is accurate where Terraform is running? Thanks. |
|
@manicminer / @tombuildsstuff Just for completness, I can confirm that the same symptom appears on v3.48 for AD creation as noted above. Around an hour mark the session expires. Here is debug log from this: It appears that, so long creation of the resource, is less than an hour there are no problems. If resource creation takes over an hour then ExpiredAuthenticationToken error will appear. |
|
Can confirm still happening for me when creating sqlmi around the hour mark also with 3.48 , using 3.43 it has no issues |
|
@manicminer |
|
@manicminer Sorry to correct my previous comment: though the ADDS is using the Meanwhile, I've noticed almost all the reports were about using azure cli to auth. Since the // Some tokens may never expire
if token.Expiry.IsZero() {
return false
}This is correctly set for client credentials: https://github.com/hashicorp/go-azure-sdk/blob/63408294321d6e958d45999dbed0105e35150e6e/sdk/auth/client_credentials.go#L343-L360 token := &oauth2.Token{
AccessToken: tokenRes.AccessToken,
TokenType: tokenRes.TokenType,
}
var secs time.Duration
if exp, ok := tokenRes.ExpiresIn.(string); ok && exp != "" {
if v, err := strconv.Atoi(exp); err == nil {
secs = time.Duration(v)
}
} else if exp, ok := tokenRes.ExpiresIn.(int64); ok {
secs = time.Duration(exp)
} else if exp, ok := tokenRes.ExpiresIn.(float64); ok {
secs = time.Duration(exp)
}
if secs > 0 {
token.Expiry = time.Now().Add(secs * time.Second)
}While seems not for the CLI auth: https://github.com/hashicorp/go-azure-sdk/blob/63408294321d6e958d45999dbed0105e35150e6e/sdk/auth/azure_cli_authorizer.go#L84-L87 So that is probably the reason why token by CLI auth always expire (and the time that users reports are a bit of random)? |
|
@magodo Thanks that's a great catch, and might explain this issue for those using Azure CLI. Either way, that's definitely a bug and one I'll fix shortly. |
|
@manicminer what's the status of the changes to the SDK and this issue? |
|
We'll be merging a fix for Azure CLI authentication in today's release which we're hoping will resolve this for those using that authentication method. Further testing for other authentication methods is still ongoing. |
|
x-ref: #21583 |
|
@katbyte I can see you have update milestones, since v.3.54.0 is released; Is ExpiredAuthenticationToken issue resolved ? I have some production deployments are on hold because of this. @manicminer - Any update from your end ? |
|
@saujanyasoni This is a complex issue which may have more than one root cause. As mentioned earlier, we released a fix for Azure CLI authentication yesterday in v3.54.0 which may resolve the issue for those using that authentication method. However, other potential causes remain under investigation and so this issue is being kept open until we are confident it has been fixed. Please continue to test and report back, as this is the only way we can get meaningful feedback. Thanks! |
|
@manicminer - It seems issue resolved for me after latest release. However other testers should also test this to be confirmed. |
@LaurentLesle It seems you are authenticating with az cli through service principal -- I didn't know that was supported? https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/guides/azure_cli
Would be great if the team could enable this 😊 This remediates the requirement for setting credentials through environment variables. |
|
Hi @cveld no I use the ARM_CLIENT_ID, ARM_CLIENT_SECRET to deploy with service principal. |
|
@LaurentLesle Are you still seeing this issue with long-running operations? In an earlier comment you stated you were authenticating via Azure CLI - would you be able to confirm whether you are seeing this and if you are authenticating via az-cli or using the provider's native support for service principal authentication? Thanks! |
|
I can confirm it is now working with the version 3.55.0. Thanks all for the fix. |
|
👍 thanks for the confirmation @LaurentLesle - since this appears to be resolved I'm going to close this issue for the moment, if you're still seeing this issue with the latest version of the Provider then please feel free to open a new issue and we can take another look. Thanks! |
|
getting same issue with |
|
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. |
Is there an existing issue for this?
Community Note
Terraform Version
1.3.9
AzureRM Provider Version
3.46.0
Affected Resource(s)/Data Source(s)
azurerm_active_directory_domain_service
Terraform Configuration Files
The text was updated successfully, but these errors were encountered: