1.3.8 merge_dicts() 'IndexError: list index out of range'

[boogi]

Description :
In version 1.3.8 I get error with the same config that works in 1.3.7

! ERROR: Hook 'load_terraform_data' from /usr/local/lib/python3.6/site-packages/terraform_compliance/steps/terrain.py:8 raised: 'IndexError: list index out of range'

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/radish/hookregistry.py", line 132, in call
    func(model, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/terraform_compliance/steps/terrain.py", line 10, in load_terraform_data
    world.config.terraform = TerraformParser(world.config.user_data['plan_file'])
  File "/usr/local/lib/python3.6/site-packages/terraform_compliance/extensions/terraform.py", line 44, in __init__
    self.parse()
  File "/usr/local/lib/python3.6/site-packages/terraform_compliance/extensions/terraform.py", line 403, in parse
    self._mount_references()
  File "/usr/local/lib/python3.6/site-packages/terraform_compliance/extensions/terraform.py", line 343, in _mount_references
    merge_dicts(self.resources[t_r]['values'][key], value)
  File "/usr/local/lib/python3.6/site-packages/terraform_compliance/common/helper.py", line 579, in merge_dicts
    merge_dicts(source[i], target[i])
  File "/usr/local/lib/python3.6/site-packages/terraform_compliance/common/helper.py", line 594, in merge_dicts
    merge_dicts(source[key], target[key])
  File "/usr/local/lib/python3.6/site-packages/terraform_compliance/common/helper.py", line 578, in merge_dicts
    if isinstance(value, dict) and isinstance(source[i], dict): # what if i not in source?
IndexError: list index out of range

To Reproduce
Plan/Code cannot be shared, but it's a valid plan and can be terraform applied fine.

It fails on just first test applied (so doesn't depend on kind of test, but only can't read plan code properly - I used "-f" switch to run different tests), e.g. below test for kms fails with this error even that there is no kms objects in our terraform code:

Feature: KMS keys should have auto-rotation enabled
  In order to improve security
  As engineers
  We'll enable key auto-rotation for every KMS key
  Based on tfsec AWS019 check (see https://github.com/liamg/tfsec#included-checks)

  Scenario: KMS keys should have auto-rotation enabled
    Given I have aws_kms_key defined
    Then enable_key_rotation must be enabled

Tested versions :

• terraform-compliance: 1.3.7 works, 1.3.8 shows error above
• terraform v0.13.5.
• not running as python package, but for reference python version is 3.6.8

Additional context
Will try to provide example if I can strip corporate code that it doesn't contain sensitive information.
Probably all is needed is the answer to "# what if i not in source?" question from line 578 :)

[boogi]

Just noting part of simple debugging, probably note mainly to myself.

Adding:

diff helper.py helper-orig.py 
578,584c578,581
<             try:
<                 if isinstance(value, dict) and isinstance(source[i], dict): # what if i not in source?
<                     merge_dicts(source[i], target[i])
<                 elif isinstance(value, list) and isinstance(source[i], list):
<                     merge_dicts(source[i], target[i])
<             except:
<                 console_write('\nEXCEPTION\ni: {}\nvalue: {}\ntarget: {}\nsource: {}\n\n'.format(i, value, target, source))
---
>             if isinstance(value, dict) and isinstance(source[i], dict): # what if i not in source?
>                 merge_dicts(source[i], target[i])
>             elif isinstance(value, list) and isinstance(source[i], list):
>                 merge_dicts(source[i], target[i])

produces:

EXCEPTION
i: 0
value: {'cookies': [{'forward': 'none'}], 'headers': ['Accept-Encoding', 'Access-Control-Request-Headers', 'Access-Control-Request-Method', 'Origin'], 'query_string': True}
target: [{'cookies': [{'forward': 'none'}], 'headers': ['Accept-Encoding', 'Access-Control-Request-Headers', 'Access-Control-Request-Method', 'Origin'], 'query_string': True}]
source: []

[Kudbettin]

@boogi

Thanks for submitting the issue.

The problem might quickly be solved by adding the tail of the target list into the source list. However, I'm hesitating to add that without reproducing the issue.

Could you try to make an anonymized plan or tf file that can be used to reproduce this error? I had trouble coming up with something that breaks.

(You can also install #440 if you'd like to see if it fixes your problem right away)

[sixdaysandy]

@Kudbettin Hi I just wanted to add that we've been having this issue with 1.3.8 and that #440 does fix the problem in our instance.

[Kudbettin]

Great!

[eerkunt]

Just released 1.3.9 having this fix. Since it has been already tested and approved, closing this issue. Please do not hesitate to open a new one if the problem persists :)

Thanks for the issue 🎉