aws cloudformation package breaks certain string literals in templates

[idm-ryou]

CloudFormation template

template.yml

Parameters:
  Value7:
    Type: String
    Default: "077777777777"
  Value8:
    Type: String
    Default: "088888888888"
  Value9:
    Type: String
    Default: "099999999999"
Resources:
  NestedStack:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: ./template_child.yml

template_child.yml

# This template is not so important
Resources:
  Dummy:
    Type: Custom::Whatever
    Properties:
      ServiceToken: whatever
      Value1: whatever

Commands to reproduce

$ uname -a
Darwin xxxxxxx 18.2.0 Darwin Kernel Version 18.2.0: Thu Dec 20 20:46:53 PST 2018; root:xnu-4903.241.1~1/RELEASE_X86_64 x86_64

$ aws --version
aws-cli/1.16.110 Python/3.7.2 Darwin/18.2.0 botocore/1.12.100

$ aws cloudformation package --s3-bucket BUCKET_NAME --template-file template.yml                                                           
Uploading to 137ce8b72427772da39a43ddc087908a.template  101 / 101.0  (100.00%)
Parameters:
  Value7:
    Type: String
    Default: '077777777777'
  Value8:
    Type: String
    Default: 088888888888
  Value9:
    Type: String
    Default: 099999999999
Resources:
  NestedStack:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: https://s3.amazonaws.com/BUCKET_NAME/137ce8b72427772da39a43ddc087908a.template

Quotes of Value8 and Value9 are removed after aws cloudformation package.

It seems that leads to the values interpreted as numbers during stack operation.
If I use the values for !Sub, values will be like 9.9999999999E10, which is unintended and cause errors.

[idm-ryou]

https://github.com/aws/aws-cli/blob/develop/awscli/customizations/cloudformation/yamlhelper.py

$ python3
Python 3.7.2 (default, Feb 12 2019, 08:15:36)
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>> from awscli.customizations.cloudformation.yamlhelper import yaml_parse, yaml_dump

>>> parsed = yaml_parse('sevens: "077777777777"\neights: "088888888888"')

>>> parsed
OrderedDict([('sevens', '077777777777'), ('eights', '088888888888')])

>>> yaml_dump(parsed)
"sevens: '077777777777'\neights: 088888888888\n"

It seems quotes are erased during dumping to yaml.

$ python3
Python 3.7.2 (default, Feb 12 2019, 08:15:36)
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import yaml

>>> yaml
<module 'yaml' from '/usr/local/Cellar/awscli/1.16.110/libexec/lib/python3.7/site-packages/yaml/__init__.py'>

>>> yaml.dump({"sevens": "077777777777", "eights": "088888888888"}, default_flow_style=False)

"eight: 088888888888\nseven: '077777777777'\n"

Seems a bug of PyYAML?:thinking:

[idm-ryou]

I thought PyYAML generates invalid yaml, but PyYAML parses correctly (as string type) without quotes.

>>> yaml.load("eights: 088888888888\nsevens: '077777777777'\n")
{'eights': '088888888888', 'sevens': '077777777777'}

I suspect either PyYAML or the yaml parser used in CloudFormation API is not following the yaml spec (related to octal base int type and string type without quotes) and causing this issue.

https://yaml.org/type/int.html

In any case, aws cloudformation package should generates yaml that can be used for CloudFormation.

[idm-ryou]

Any updates?
Please let me know if you need further information.

[idm-ryou]

I guessed that CloudFormation API uses SnakeYAML from its behavior, so I filed an issue for SnakeYAML.

https://bitbucket.org/snakeyaml/snakeyaml/issues/442

The SnakeYAML author says that it is a bug in YAML 1.1 spec...

I think aws-cli should dump strings in YAML with quotes for parser interoperability.
https://stackoverflow.com/questions/38369833/pyyaml-and-using-quotes-for-strings-only

[idm-ryou]

Related?
yaml/pyyaml#98
awslabs/aws-cfn-template-flip#41
aws/aws-sam-cli#867

[justnance]

We are looking into this issue. @sanathkr, what are your thoughts?

[beibeiyang]

We are running into a similar issue. We have Account ID as the key in template.yaml:

template.yaml

...
Mappings:
  AccountMap:
    "098111111198":
      Env: "dev"
    "988111111198":
      Env: "test"
    "123456789012":
      Env: "prod"
...

After the sam package --template-file template.yaml --output-template-file packaged.yaml --s3-bucket BUCKET_NAME, quotes around the first Account ID got stripped:

packaged.yaml

...
Mappings:
  AccountMap:
    098111111198:
      Env: dev
    '988111111198':
      Env: test
    '123456789012':
      Env: prod
...

The subsequent aws cloudformation deploy --template-file packaged.yaml --stack-name STACK_NAME ... thinks 098111111198 is a number and throws An error occurred (ValidationError) when calling the CreateChangeSet operation: Template format error: [/Mappings/AccountMap] map keys must be strings; received numeric [9.8111111198E10] instead.

$ sam --version
SAM CLI, version 0.17.0
$ aws --version
aws-cli/1.16.150 Python/3.7.3 Darwin/18.5.0 botocore/1.12.140
$ python
Python 3.6.7 | packaged by conda-forge | (default, Feb 28 2019, 02:16:08)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> boto3.__version__
'1.9.172'

We would appreciate suggestions to any workarounds. Thanks!

[jiravani]

Having the exact same issue outlined by @beibeiyang above. Any updates?

[bs-thomas]

I'm experiencing the same weird issue. Very weird. Things up to 7 are working, and 8 will break as per above issue.

[nmnielsen]

I've also encountered the same issue (awscli 1.16.199), any updates or plans to produce updates?

[concordion2k]

Same issue here, when following instructions to create a bucket policy for us-east-2, using was-cli 1.16.260:

https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html#enable-access-logging

[idm-ryou]

I've switched to use AWS CDK after all.
CDK is not affected by this issue, I suppose.

[bamapookie]

@justnance Is this still being worked on? Is there a workaround?

[gauravlanjekar]

This breaks the creation of Cognito UserPool because the MFAConfiguration requires quotes surrounding the "OFF" value these are stripped off by the cli . Is there any workaround to force quotes ?

[lmayorga1980]

@idm-ryou Did you find a workaround in the sam template for account_id map lookup?

[bamapookie]

@lmayorga1980 I found a workaround. I had to move the reference to the offending numerical string to the top level template file (since I was using it in a nested stack), and then add --use-json to the `aws cloudformation package command. This will cause the top level stack to be output as json instead of yaml.

[375gnu]

Today I faced strange error with templates and parameters which were working without any issues:

An error occurred (ValidationError) when calling the CreateStack operation: Parameter 'EnableHA' must be one of AllowedValues

This EnableHA has only two allowed values: "yes" and "no", and my parameters json file defines it to "no".

Investigation showed that aws-cli 2 (2.0.7 and 2.0.9) converts these strings to just yes and no which have special meaning in YAML and so my supplied parameters don't pass validation.

This should be fixed ASAP.

[JamesDougherty]

We were having the same problem in our SAM template since our account starts with a '0' as well. I was able to get around it by using !Ref 'AWS::AccountId' where I needed the account ID.

See here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/pseudo-parameter-reference.html

[tthyer]

I have just run into the same issue when trying to stand up one AWS genomics' cloudformation stacks. It uses a parameter of type String whose AllowedValues are Yes and No. When I attempt to pass one of those strings using awscli2, I get a message that I must use one of the AllowedValues. It doesn't matter whether the value is quote when I pass the ParameterValue. Interestingly, when I run validate-template, this is in the output:

{
            "ParameterKey": "ExistingBucket",
            "DefaultValue": "false",
            "NoEcho": false,
            "Description": "Does this bucket already exist?"
}

So it appears that at some point yes and no become true and false. The workaround is to just pass the strings true or false.

[JorisLimousinKaizen]

Experiencing the same issue with account IDs as Map keys.

[shinglyu]

This causes problems for a string with colons like so:

Resources:
  WindowsFileSystemWithAllConfigs:
    Type: "AWS::FSx::FileSystem"
    # ...
      WindowsConfiguration:
        WeeklyMaintenanceStartTime: "4:16:30"
        DailyAutomaticBackupStartTime: "01:00"
        # ...

It will be come WeeklyMaintenanceStartTime: 4:16:30, which is then interpreted as a number and violates the field validation rule.

Example error while deploying:

2 validation errors detected: Value '15390' at 'windowsConfiguration.weeklyMaintenanceStartTime' failed to satisfy constraint: Member must satisfy regular expression pattern: ^[1-7]:([01]\d|2[0-3]):?([0-5]\d)$; Value '15390' at 'windowsConfiguration.weeklyMaintenanceStartTime' failed to satisfy constraint: Member must have length greater than or equal to 7 (Service: AmazonFSx; Status Code: 400; Error Code: BadRequest; Request ID: cbeff8a0-ce03-452c-9f0a-f8e08d894dc9; Proxy: null)

[seren]

We’re running into this on our end as well. It's surprising and concerning that this issue is 2 years old is still unaddressed.

[tjtaill]

I just ran into this issue to seems sporadic as well was working and all of a sudden stopped working on a template that was working

[ConnorKirk]

I have encountered the same issue.

I can replicate this issue on v 1.19.97 of the CLI
I cannot replicate it on v2.2.12

[SonyDavid]

Encounter this issue in aws-cli/2.7.3 Python/3.9.11. The single quote from Time being remove by package for AWS::DLM::LifecyclePlicy PolicyDetails Schedules CreateRules.

Submitted Template:
CreateRule: Interval: 12 IntervalUnit: HOURS Times: - '16:00'

Produced by package command:
CreateRule: Interval: 12 IntervalUnit: HOURS Times: - 16:00

[mp24-git]

I see the issue when trying to create a custom resource for the AWS Instance Scheduler.

It works in the CodeBuild image aws/codebuild/standard:4.0 which uses:
aws-cli/1.25.1 Python/3.8.13 Linux/4.14.275-207.503.amzn2.x86_64 exec-env/AWS_ECS_EC2 botocore/1.27.1

It does not work with CodeBuild image aws/codebuild/standard:5.0 which uses:
aws-cli/2.7.5 Python/3.9.11 Linux/4.14.275-207.503.amzn2.x86_64 exec-env/AWS_ECS_EC2 exe/x86_64.ubuntu.20 prompt/off

The input looks like this:

AWSTemplateFormatVersion: '2010-09-09'
Description: AWS Instance Schedule Definition Stack
Parameters:
  ServiceToken:
    Type: String
Resources:
  EC2Schedule:
    Type: Custom::ServiceInstanceSchedule
    Properties:
      Name: EC2Schedule
      Description: Ec2 Uptime
      NoStackPrefix: 'True'
      Enforced: 'False'
      Hibernate: 'False'
      Timezone: Europe/Berlin
      ServiceToken: !Ref 'ServiceToken'
      Periods:
        - Description: End on weekdays
          EndTime: '23:00'
          WeekDays: Mon-Fri
        - Description: Core Working hours Thursday
          BeginTime: '08:00'
          EndTime: '18:00'
          WeekDays: Thu
        - Description: Weekend Maintenance Window
          BeginTime: '00:01'
          EndTime: '01:30'
          WeekDays: Sun

Output looks like this:

AWSTemplateFormatVersion: '2010-09-09'
Description: AWS Instance Schedule Definition Stack
Parameters:
  ServiceToken:
    Type: String
Resources:
  EC2Schedule:
    Type: Custom::ServiceInstanceSchedule
    Properties:
      Name: EC2Schedule
      Description: Ec2 Uptime
      NoStackPrefix: 'True'
      Timezone: Europe/Amsterdam
      ServiceToken:
        Ref: ServiceToken
      Periods:
      - Description: End on weekdays
        EndTime: 23:00
        WeekDays: Mon-Fri
      - Description: Core Working hours Thursday
        BeginTime: 08:00
        EndTime: 18:00
        WeekDays: Thu
      - Description: Weekend Maintenance Window
        BeginTime: 00:01
        EndTime: 01:30
        WeekDays: Sun