Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Concurrency control? #38

Open
iainelder opened this issue Jun 29, 2021 · 1 comment · May be fixed by #39
Open

Concurrency control? #38

iainelder opened this issue Jun 29, 2021 · 1 comment · May be fixed by #39

Comments

@iainelder
Copy link

I felt ambitious today and tried to list 17 accounts in 17 regions.

awsls failed before listing anything.

$ awsls cloudtrail --regions "${regions}" --profiles "${profiles}"


Error: failed to launch provider (/home/isme/.awsls/terraform-provider-aws_v3.31.0_x5): pipe2: too many open files

I'm not really sure what limit I'm hitting here, whether it's imposed by awsls itself or by my operating system.

When I halved the number of regions to 8, the behavior was unpredictable and unstable.

Once it froze for a while before printing:

Finished (killed)

Another time it printed something more esoteric:

panic: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"

goroutine 254 [running]:
github.com/hashicorp/terraform/plugin.(*GRPCProvider).getSchema(0xc0002a7500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/home/runner/go/pkg/mod/github.com/hashicorp/[email protected]/plugin/grpc_provider.go:80 +0x289
github.com/hashicorp/terraform/plugin.(*GRPCProvider).Configure(0xc0002a7500, 0x0, 0x0, 0x8270f98, 0xc002107060, 0x6e01b40, 0xc00209d770, 0x414494, 0xc003bb9d60, 0x6e013c0)
	/home/runner/go/pkg/mod/github.com/hashicorp/[email protected]/plugin/grpc_provider.go:277 +0x6a
github.com/jckuester/terradozer/pkg/provider.TerraformProvider.Configure(0x7fdaa9527390, 0xc0002a7500, 0x2540be400, 0x8270f98, 0xc002107060, 0x6e01b40, 0xc00209d770, 0x0, 0x0)
	/home/runner/go/pkg/mod/github.com/jckuester/[email protected]/pkg/provider/provider.go:101 +0xbc
github.com/jckuester/awstools-lib/terraform.NewProviderPool.func1(0xc0002e4750, 0xc0003d0aa5, 0x3, 0xc0003d0aaa, 0x6, 0xc0003d0a80, 0x33, 0x2540be400, 0xc000200300, 0xc00005bd90, ...)
	/home/runner/go/pkg/mod/github.com/jckuester/[email protected]/terraform/provider_pool.go:90 +0x1233
created by github.com/jckuester/awstools-lib/terraform.NewProviderPool
	/home/runner/go/pkg/mod/github.com/jckuester/[email protected]/terraform/provider_pool.go:53 +0x657

I suppose there is a certain stable limit for the number of accounts and regions I can query.

Is there some way to control the concurrency of awsls without splitting the calls into batches?
@jckuester jckuester linked a pull request Jul 10, 2021 that will close this issue
Open
@jckuester
Copy link
Owner

jckuester commented Jul 10, 2021 •
edited
Loading

Hi @iainelder, I started working on the first issues that you are describing. All provider plugins (17x17=289) are launched in the beginning and lead to too many open files. I'll try to launch/close the provider each time when needed (this is also what Terraform does and the overhead of launch/close is very small): #39

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix: Open too many provider files jckuester/awsls
2 participants
@jckuester @iainelder