Terragrunt does not support AWS SSO with automatic token refresh

[lkoniecz]

 terraform --version
Terraform v1.2.9
on darwin_arm64
+ provider registry.terraform.io/datadog/datadog v3.19.1
+ provider registry.terraform.io/hashicorp/aws v4.45.0

Your version of Terraform is out of date! The latest version
is 1.3.7. You can update by downloading from https://www.terraform.io/downloads.html

 terragrunt --version
terragrunt version v0.39.0

~/.aws/config

[profile dev-sso]
sso_session = sso
sso_start_url = https://my-company/start
sso_region = eu-west-1
sso_account_id = 123456789
sso_role_name = AWSAdministratorAccess

[sso-session sso]
sso_region = eu-west-1
sso_start_url = https://my-company/start
sso_registration_scope = sso:account:access

Remote state config

remote_state {
  backend = "s3"
  config = {
    bucket         = "${local.env}-my-bucket"
    region         = "us-east-1"
    key            = "${path_relative_to_include()}/terraform.tfstate"
    encrypt        = true
    dynamodb_table = "${local.env}-terraform-state-lock"
    profile        = "dev-sso"
  }
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
}

 terragrunt init -migrate-state                             

Initializing the backend...
Backend configuration changed!

Terraform has detected that the configuration specified for the backend
has changed. Terraform will now check for existing state in the backends.

╷
│ Error: error configuring S3 Backend: no valid credential sources for S3 Backend found.
│ 
│ Please see https://www.terraform.io/docs/language/settings/backends/s3.html
│ for more information about providing credentials.
│ 
│ Error: NoCredentialProviders: no valid providers in chain. Deprecated.
│       For verbose messaging see aws.Config.CredentialsChainVerboseErrors
│ 

╵

 aws sts get-caller-identity
{
    "UserId": "my_user_id",
    "Account": "123456789",
    "Arn": "arn:aws:sts::123456789:assumed-role-bla-bla-bla
}

[denis256]

hi,
will be helpful to post output of execution with --terragrunt-log-level debug

Additionally, can be attempted to use aws-valut to provide all required profile settings, but this approach will work only with one profile for entire execution

[lkoniecz]

 terragrunt init -migrate-state --terragrunt-log-level debug
DEBU[0000] Did not find any locals block: skipping evaluation. 
DEBU[0000] Detected single bare include block - exposing as top level 
DEBU[0000] Found locals block: evaluating the expressions. 
DEBU[0000] Evaluated 1 locals (remaining 1): env        
DEBU[0000] Evaluated 1 locals (remaining 0): aws_profile 
DEBU[0000] [Partial] Included config /Users/lky/repositories/infrastructure-as-code/terraform/envs/terragrunt.hcl has strategy shallow merge: merging config in (shallow). 
DEBU[0000] Running command: terraform --version          prefix=[/Users/lky/repositories/infrastructure-as-code/terraform/envs/dev/datadog] 
DEBU[0000] Terraform version: 1.2.9                     
DEBU[0000] Reading Terragrunt config file at /Users/lky/repositories/infrastructure-as-code/terraform/envs/dev/datadog/terragrunt.hcl 
DEBU[0000] Did not find any locals block: skipping evaluation. 
DEBU[0000] Detected single bare include block - exposing as top level 
DEBU[0000] Found locals block: evaluating the expressions. 
DEBU[0000] Evaluated 1 locals (remaining 1): env        
DEBU[0000] Evaluated 1 locals (remaining 0): aws_profile 
DEBU[0000] [Partial] Included config /Users/lky/repositories/infrastructure-as-code/terraform/envs/terragrunt.hcl has strategy shallow merge: merging config in (shallow). 
DEBU[0000] Did not find any locals block: skipping evaluation. 
DEBU[0000] Detected single bare include block - exposing as top level 
DEBU[0000] Found locals block: evaluating the expressions. 
DEBU[0000] Evaluated 1 locals (remaining 1): env        
DEBU[0000] Evaluated 1 locals (remaining 0): aws_profile 
DEBU[0000] Included config /Users/lky/repositories/infrastructure-as-code/terraform/envs/terragrunt.hcl has strategy shallow merge: merging config in (shallow) for dependency. 
DEBU[0000] Detected single bare include block - exposing as top level 
DEBU[0000] Found locals block: evaluating the expressions. 
DEBU[0000] Evaluated 1 locals (remaining 1): env        
DEBU[0000] Evaluated 1 locals (remaining 0): aws_profile 
DEBU[0000] Found locals block: evaluating the expressions. 
DEBU[0000] Evaluated 1 locals (remaining 1): env        
DEBU[0000] Evaluated 1 locals (remaining 0): aws_profile 
DEBU[0000] Included config /Users/lky/repositories/infrastructure-as-code/terraform/envs/terragrunt.hcl has strategy shallow merge: merging config in (shallow). 
WARN[0000] No double-slash (//) found in source URL /Users/lky/repositories/infrastructure-as-code/terraform/datadog. Relative paths in downloaded Terraform code may not work. 
DEBU[0000] Terraform files in /Users/lky/repositories/infrastructure-as-code/terraform/envs/dev/datadog/.terragrunt-cache/yl9oT0rFY0GdMyu8dU5Rjho8bdo/Hx_7vqx9v9m8e2qrsgz0ACVqOM0 are up to date. Will not download again. 
DEBU[0000] Copying files from /Users/lky/repositories/infrastructure-as-code/terraform/envs/dev/datadog into /Users/lky/repositories/infrastructure-as-code/terraform/envs/dev/datadog/.terragrunt-cache/yl9oT0rFY0GdMyu8dU5Rjho8bdo/Hx_7vqx9v9m8e2qrsgz0ACVqOM0 
DEBU[0000] Setting working directory to /Users/lky/repositories/infrastructure-as-code/terraform/envs/dev/datadog/.terragrunt-cache/yl9oT0rFY0GdMyu8dU5Rjho8bdo/Hx_7vqx9v9m8e2qrsgz0ACVqOM0 
DEBU[0000] The file path /Users/lky/repositories/infrastructure-as-code/terraform/envs/dev/datadog/.terragrunt-cache/yl9oT0rFY0GdMyu8dU5Rjho8bdo/Hx_7vqx9v9m8e2qrsgz0ACVqOM0/backend.tf already exists, but was a previously generated file by terragrunt. Since if_exists for code generation is set to "overwrite_terragrunt", regenerating file.  prefix=[/Users/lky/repositories/infrastructure-as-code/terraform/envs/dev/datadog] 
DEBU[0000] Generated file /Users/lky/repositories/infrastructure-as-code/terraform/envs/dev/datadog/.terragrunt-cache/yl9oT0rFY0GdMyu8dU5Rjho8bdo/Hx_7vqx9v9m8e2qrsgz0ACVqOM0/backend.tf.  prefix=[/Users/lky/repositories/infrastructure-as-code/terraform/envs/dev/datadog] 
DEBU[0000] Backend config changed from map[access_key:<nil> acl:<nil> assume_role_duration_seconds:<nil> assume_role_policy:<nil> assume_role_policy_arns:<nil> assume_role_tags:<nil> assume_role_transitive_tag_keys:<nil> bucket:***** dynamodb_endpoint:<nil> dynamodb_table:dev-terraform-state-lock encrypt:%!s(bool=true) endpoint:<nil> external_id:<nil> force_path_style:<nil> iam_endpoint:<nil> key:dev/datadog/terraform.tfstate kms_key_id:<nil> max_retries:<nil> profile:dev-sso region:us-east-1 role_arn:<nil> secret_key:<nil> session_name:<nil> shared_credentials_file:<nil> skip_credentials_validation:<nil> skip_metadata_api_check:<nil> skip_region_validation:<nil> sse_customer_key:<nil> sts_endpoint:<nil> token:<nil> workspace_key_prefix:<nil>] to map[bucket:***** dynamodb_table:dev-terraform-state-lock encrypt:%!s(bool=true) key:dev/datadog/terraform.tfstate profile:dev-sso region:us-east-1]  prefix=[/Users/lky/repositories/infrastructure-as-code/terraform/envs/dev/datadog] 
DEBU[0000] Initializing remote state for the s3 backend  prefix=[/Users/lky/repositories/infrastructure-as-code/terraform/envs/dev/datadog] 
ERRO[0000] Error finding AWS credentials (did you set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables?): SSOProviderInvalidToken: the SSO session has expired or is invalid 
ERRO[0000] Unable to determine underlying exit code, so Terragrunt will exit with error code 1 

[lordz-ei]

Any info on this?
I have a similar config in ~/.aws/coinfig

[profile sandbox]
sso_session = TraianMac
sso_account_id = 1234567890
sso_role_name = LimitedAdministratorAccess
region = eu-west-1
output = json

[sso-session TraianMac]
sso_start_url = https://somecool.awsapps.com/start#/
sso_region = eu-west-1
sso_registration_scopes = sso:account:access

and when i try to run terragrunt plan I get

DEBU[0000] Included config /Users/traian/EI_Code/infrastructure-environments/dm-sandbox/terragrunt.hcl has strategy shallow merge: merging config in (shallow).
DEBU[0000] Assuming IAM role arn:aws:iam::1234567890:role/terragrunt-role with a session duration of 0 seconds.
ERRO[0000] profile "sandbox" is configured to use SSO but is missing required configuration: sso_region, sso_start_url
ERRO[0000] Unable to determine underlying exit code, so Terragrunt will exit with error code 1

[hegarty-poloniex]

I'm facing the same issue running terragrunt version v0.38.4, however I quickly discovered that manually updating ~/.aws/config to use the legacy format will work as expected. Personally I always manually updated ~/.aws/config anyway, as opposed to using the aws configure sso wizard. That said we need to see this solved...

[profile dev-workload-1].
sso_start_url = https://d-0000000000.awsapps.com/start.
sso_region = eu-west-1.
sso_account_id = 0000000000.
sso_role_name = role-name.
region = eu-west-1

[knmsk]

I've used the latest 0.45.2(at the time of writing this) in an ARM64 Macbook and got the same error, but I got it running after cloning the repository and make build after changing the $$(git describe --tags --abbrev=12 --dirty --broken) from the makefile to a random string.

I hope this helps someone

$ terragrunt init                    

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
Initializing modules...
- private_subnets in modules/aws-private-subnets
- public_subnets in modules/aws-public-subnets

Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Installing hashicorp/aws v4.62.0...
- Installed hashicorp/aws v4.62.0 (signed by HashiCorp)

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

[aljoshare]

Does somebody know, if Terragrunt or Terraform is doing the sso stuff? Because there are similar issues in the AWS provider repository and the Terraform repository. Maybe it will be solved automatically, when both are closed. Pretty shitty situation because a lot of stuff is broken since the sso configuration change.

[lordz-ei]

Does somebody know, if Terragrunt or Terraform is doing the sso stuff? Because there are similar issues in the AWS provider repository and the Terraform repository. Maybe it will be solved automatically, when both are closed. Pretty shitty situation because a lot of stuff is broken since the sso configuration change.

Terragrunt is a wrapper over terraform so there is a chance that this will be solved when that issues will be fixed

[LozanoMatheus]

I'm getting a similar error as @lkoniecz and I seems to be a terragrunt issue, it works just fine with terraform. Also, the terragrunt seems to require two (duplicated) parameters, the sso_region and sso_start_url even tho they're already configured under the [sso-session <sso_session>].

When I try to run (terraform==1.4.6, terragrunt==0.45.9) I get the error below

ERRO[0000] Error finding AWS credentials (did you set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables?): SSOProviderInvalidToken: the SSO session has expired or is invalid
caused by: open /home/<my_user>/.aws/sso/cache/<hash001>.json: no such file or directory
ERRO[0000] Unable to determine underlying exit code, so Terragrunt will exit with error code 1

So, what I did was to run the AWS CLI commands to (re)configure the local stuff

aws configure sso --profile <my_profile> --region eu-west-1
aws sso login --profile <my_profile>  --region eu-west-1 --sso-session <my_sso_session> 

Then I checked which files I had under the ~/.aws/sso/cache/ and I found out it wasn't the same as the <hash001>.json, so I create a symbolic link from the one I've <hash002>.json to <hash001>.json.

ln -s /home/<my_user>/.aws/sso/cache/<hash002>.json /home/<my_user>/.aws/sso/cache/<hash001>.json

Then I tried to rerun terragrunt and got a different error which points to a third file (<hash003>.json).

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found.
│
│ Please see https://registry.terraform.io/providers/hashicorp/aws
│ for more information about providing credentials.
│
│ AWS Error: failed to refresh cached credentials, failed to read cached SSO token file, open /home/<my_user>/.aws/sso/cache/<hash003>.json: no such file or directory
│
│
│   with provider["registry.terraform.io/hashicorp/aws"],
│   on providers.tf line 2, in provider "aws":
│    2: provider "aws" {
│
╵
ERRO[0004] Terraform invocation failed in /home/<my_user>/<path>/<to>/<my>/<tf_code>
ERRO[0004] 1 error occurred:
        * exit status 1

So, to so this I create another symbolic link from <hash002>.json to <hash003>.json

ln -s /home/<my_user>/.aws/sso/cache/<hash002>.json /home/<my_user>/.aws/sso/cache/<hash003>.json

Now everything works just fine, but I'm not sure if these ~/.aws/sso/cache files are created with a different name at some point, if so, then I'll have to create the symbolic links again.

---

This is what I've under my ~/.aws/config

[profile <my_profile>]
sso_region = <my_sso_region>
sso_session = <my_sso_session>
sso_account_id = <my_sso_account_id>
sso_role_name = <my_sso_role_name>
region = <my_region>
sso_start_url = <my_sso_start_url>

[sso-session <my_sso_session>]
sso_start_url = <my_sso_start_url>
sso_region = <my_sso_region>
sso_registration_scopes = sso:account:access

[levkohimins]

The same issue is mentioned in terraform: #28263 #32465. Since terragrunt runs terraform as a shell command under the hood it can do nothing until hashicorp team fixes it.

But as discussed in those issues, there is a workaround. Step by step to make it work (I will use as session name: my-session and as profile name: my-profile):

1. Just in case, remove the session and profile sections from ~/.aws/config that you are about to configure if they are already there.

2. Run

aws configure sso --profile my-profile

The above command adds the following sections to ~/.aws/config:

[profile my-profile]
sso_session = my-session
sso_account_id = xxxxxxxxxxxx
sso_role_name = xxxxxxx
[sso-session my-session]
sso_start_url = https://xxxxxx.awsapps.com/start#
sso_region = us-east-1
sso_registration_scopes = sso:account:access

Let's check SSO in aws-cli, it should work

aws s3 ls --profile my-profile

Then try to check SSO in terraform

AWS_PROFILE=my-profile terraform init

get:

Error: error configuring S3 Backend: Error creating AWS session: profile "my-profile" is configured to use SSO but is missing required configuration: sso_region, sso_start_url

don't worry, we're not done yet 🙂

3. Manually edit ~/.aws/config:

Add sso_start_url, sso_region and remove sso_session. Should look like this:

[profile my-profile]
sso_account_id = xxxxxxxxxxxx
sso_role_name = xxxxxxx
region = us-east-1
sso_start_url = https://xxxxxx.awsapps.com/start#
sso_region = us-east-1

4. Run

aws sso login --profile my-profile

Let's try to check SSO in terraform again:

AWS_PROFILE=my-profile terraform init

get:

... omitted
Terraform has been successfully initialized!
... omitted

Finally it works 🎉 . If SSO works in terraform, then it will work in terragrunt.

[jsimoni]

The same issue is mentioned in terraform: #28263 #32465. Since terragrunt runs terraform as a shell command under the hood it can do nothing until hashicorp team fixes it.

But as discussed in those issues, there is a workaround. Step by step to make it work (I will use as session name: my-session and as profile name: my-profile):

1. Just in case, remove the session and profile sections from ~/.aws/config that you are about to configure if they are already there.
2. Run

aws configure sso --profile my-profile

The above command adds the following sections to ~/.aws/config:

[profile my-profile]
sso_session = my-session
sso_account_id = xxxxxxxxxxxx
sso_role_name = xxxxxxx
[sso-session my-session]
sso_start_url = https://xxxxxx.awsapps.com/start#
sso_region = us-east-1
sso_registration_scopes = sso:account:access

Let's check SSO in aws-cli, it should work

aws s3 ls --profile my-profile

Then try to check SSO in terraform

AWS_PROFILE=my-profile terraform init

get:

Error: error configuring S3 Backend: Error creating AWS session: profile "my-profile" is configured to use SSO but is missing required configuration: sso_region, sso_start_url

don't worry, we're not done yet 🙂

3. Manually edit ~/.aws/config:

Add sso_start_url, sso_region and remove sso_session. Should look like this:

[profile my-profile]
sso_account_id = xxxxxxxxxxxx
sso_role_name = xxxxxxx
region = us-east-1
sso_start_url = https://xxxxxx.awsapps.com/start#
sso_region = us-east-1

4. Run

aws sso login --profile my-profile

Let's try to check SSO in terraform again:

AWS_PROFILE=my-profile terraform init

get:

... omitted
Terraform has been successfully initialized!
... omitted

Finally it works 🎉 . If SSO works in terraform, then it will work in terragrunt.

@levkoburburas will terraform/terragrunt automatically refresh the authentication tokens when they expire? For reference: https://docs.aws.amazon.com/cli/latest/userguide/sso-configure-profile-token.html

[levkohimins]

@levkoburburas will terraform/terragrunt automatically refresh the authentication tokens when they expire? For reference: https://docs.aws.amazon.com/cli/latest/userguide/sso-configure-profile-token.html

@jsimoni if you mean aws sso login --sso-session my-sso unfortunately it won't. Consider using aws-vault.

[lebenitza]

This should be reopened. The issue in terraform is now fixed in Terraform v1.6.0-rc1 but the issue is still present in terragrunt. You can test that by using a generate block instead of a remote_state. When using generate terraform won't complain about SSO but it is going to complain the S3 bucket does not exist. In order to let terragrunt take care of creating the backend resources you need to switch to using a remote_state and that is when the issue appears.

For me that means there is a problem in the terragrunt auth mechanism with AWS in the bit of code that makes sure the backend resources are created.

❯ terragrunt init
ERRO[0000] Error initializing session: profile "<redacted> is configured to use SSO but is missing required configuration: sso_region, sso_start_url
ERRO[0000] Unable to determine underlying exit code, so Terragrunt will exit with error code 1

❯ terragrunt version
Terraform v1.6.0-rc1
on darwin_arm64
+ provider registry.terraform.io/cloudflare/cloudflare v4.15.0
+ provider registry.terraform.io/hashicorp/aws v5.19.0

❯ terragrunt --version
terragrunt version 0.51.6

The workaround still works by moving sso_start_url and sso_region to the profile entry in aws config.
For more details see: hashicorp/terraform#32465

[WesselAtWork]

Still an issue with terragrunt.

terragrunt init        
time=TZ level=error msg=Error initializing session: profile "<child-profile>" is configured to use SSO but is missing required configuration: sso_region, sso_start_url
time=TZ level=error msg=Unable to determine underlying exit code, so Terragrunt will exit with error code 1

But when I drop into the cache directory, all works fine with terraform.

cd ./.terragrunt-cache/<HASH>/<HASH>
terraform init
...
terraform plan
...
terraform apply

terragrunt
terragrunt version v0.45.15

terraform version
Terraform v1.6.2
on windows_amd64
+ provider registry.terraform.io/hashicorp/aws v5.22.0

At first I thought it might be the outdated winget terragrunt package,
But testing on the newest release results in the same:

terragrunt --version
terragrunt version v0.52.4

terragrunt init        
time=TZ level=error msg=Error initializing session: profile "<child-profile>" is configured to use SSO but is missing required configuration: sso_region, sso_start_url
time=TZ level=error msg=Unable to determine underlying exit code, so Terragrunt will exit with error code 1

I now suspect it's something inside of terragrunt.
Maybe in the aws_helper ?

terragrunt/aws_helper/config.go

Lines 78 to 83 in b8cf75f

	sess, err := session.NewSessionWithOptions(sessionOptions)
	if err != nil {
	return nil, errors.WithStackTraceAndPrefix(err, "Error initializing session")
	}

Removing my remote_state config (that uses a s3 bucket configured with a sso profile) fixes terragrunt
Also running the remote_state config with a profile that doesn't use sso allows terragrunt to function.

[lebenitza]

Still an issue with terragrunt.

S3 backends should be fixed with the terraform 1.6 upgrade

Please see my message.... The issue is still in terragrunt with the automatic creation of backend S3 store and DynamoDB lock table.

[levkohimins]

Hi @lebenitza, Ah you are right, terragrunt uses AWS for the things you mentioned. Since this issue has been fixed in terraform, it makes sense to fix it in terragrung as well. The AWS library should be updated in terragrunt to support SSO, I'll work on it.

[lebenitza]

Yes you need to wait for terragrunt to merge support for terraform 1.6 in the issue I linked..... I'm pointing you to the in progress fix. Please read

That issue was closed 14 hours ago and its focus was in no way SSO syntax support, which was a problem hidden by the fact terraform did not support it until 1.6.0 either, making everyone believe there is nothing else to do in terragrunt code for that specific area.

Than you @levkohimins . Let me know if I can help futher.

[levkohimins]

Resolved in v0.53.1 release.

[WesselAtWork]

Just want to note it's working for me.
Running v0.53.2
Thanks @levkohimins !

[levkohimins]

My pleasure, @WesselAtWork

[Striar-Yunis]

Is this fully fixed for DRY args? The following does not work:

terraform {
  source = "foo"

  before_hook "before_hook" {
    commands = [
      "init",
      "apply",
      "refresh",
      "import",
      "plan",
      "taint",
      "untaint"
    ]

    execute = [
      "aws",
      "sso",
      "login",
      "--profile",
      "bar"
    ]
  }

  extra_arguments "force_profile" {
    commands = [
      "init",
      "apply",
      "refresh",
      "import",
      "plan",
      "taint",
      "untaint"
    ]
    arguments = []
    env_vars = {
      AWS_PROFILE = "bar"
    }
  }
}

$ terragrunt plan
 ERROR  Error finding AWS credentials (did you set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables?): NoCredentialProviders: no valid providers in chain. Deprecated.

The following does work:

AWS_PROFILE=bar terragrunt plan

This seems like a bug.