Add support --recursive option

[wata727]

Fixes #527

This is a PoC of recursive directory scanning. I've heard that for large projects there are cases where you want to run tflint in a different context than terraform apply. In such cases, it doesn't make us happy to maintain a shell script that walks directories and runs tflint in each directory.

Run with the --recursive option to walk each directory and run the analysis:

$ tflint --recursive

However, recursive mode has some limitations:

• Module inspection is not allowed
• Named values like terraform.* and path.* are ignored

These are necessary to run in a different context than Terraform. However, I'm not sure if it can solve the problems of users with complex project structures when there are these limitations. I welcome feedback on this matter.

[bendrucker]

I've encountered this problem before and have typically included Terraform specific logic in the directory finding, usually checking for *.tf and possibly backend.tf for validation that is reserved for root modules.

Given that multiple tools are likely to be involved in validation (e.g., terraform validate, tflint) I am always surprised to hear people assert that individual tools like TFLint should handle this. I don't deny that it's a recurring problem, but it seems to me like a meta-issue when working in Terraform monorepos. I've thought about writing a CLI to help but have never gotten around it.

[wata727]

Thank you for your comment.

Yes, I'm also wondering if this problem should be fixed with TFLint. Probably, users also have similar issues with terraform validate.
I guess It may be good to hear a little more about the voice of the user who working with Terraform monorepos...

[bendrucker]

I currently work in an active monorepo at @TakeScoop FWIW, and we just have a shell script handling this at the moment. The landscape for Terraform static validation is a little patchy, particularly with some rough edges in terraform validate around provider config requirements. I would imagine a lot of users have shell scripts, and that each tool increases the complexity a fair amount.

[djakielski]

Dose it also work with modules and submodules? This will be very important for me, because my code are structured like java packages by using modules.

[bendrucker]

Please build and try this on your project, hard to comment without a reproducible example.

[bendrucker]

A related thought I've had that might be relevant is looking upward for config and merging, like git does with configuration files like gitignore. This would allow for a top level TFLint configuration in the repo root, overrides for specific modules in their directories, and make it viable to run tflint in a deeply nested directory without having to pass config.

[bendrucker]

That change conflicts a bit with existing config handling but a WIP that I tested against a real monorepo worked as expected:

https://github.com/terraform-linters/tflint/tree/walk-up-config

[djakielski]

Here is some example how i use terraform

main.tf
  modules
    featureOrTopicA
      ec2.tf
    featureOrTopicB
      apiGateway.tf
    featureOrTopic...
      lambdas.tf

In Main.tf there are just instance of modules

module "featureOrTopicA" {
   source ="./modules/featureOrTopicA"
   //... attributes
}

module "featureOrTopicB" {
   source ="./modules/featureOrTopic..."
   //... attributes
}

module "featureOrTopic..." {
   source ="./modules/featureOrTopic..."
   //... attributes
}

for me it's the best way to structure compoments of terraform code.

[bendrucker]

That example would be a good fit for TFLint as it exists today. If you invoke TFLint on the root directory with --module, it will follow the module calls and evaluate each of those modules. Using the recursive option along with --module would result in duplicate runs for each module in modules/. Seems like a significant opportunity for additional confusion.

This feature would be needed for cases where you have lots of modules that don't share a Terraform entrypoint. In your example, if main.tf did not exist, it would allow you to validate your modules/ dir.