[AVM Module Issue]: Deployment Concurrency issue with Private Endpoints

[tyconsulting]

Check for previous/existing GitHub issues

• I have checked for previous/existing GitHub issues

Issue Type?

Bug

Module Name

Other, as defined below...

(Optional) Module Name if not listed above

All modules calling the Private Endpoint module

(Optional) Module Version

No response

Description

The Private Endpoint modules are called in a for loop. This can potentially cause issues when more than 1 Private Endpoints are created at the same time:

To work around issue, I'm suggesting use the @batchSize decorator to limit to 1 concurrent deployment in all modules calling the Private Endpoint module. for example:

@batchSize(1)
module workspace_privateEndpoints 'br/public:avm/res/network/private-endpoint:0.3.3' = [for (privateEndpoint, index) in (privateEndpoints ?? []): {
...
}

I have tested this in my lab and I can confirm it fixed the error I have shown above.

cc @segraef @AlexanderSehr

(Optional) Correlation Id

No response

[AlexanderSehr]

Hmm that's new. A batchSize would surely solve the issue - but it's a shame given that it slows down the deployment. Ah well...
@segraef I guess we could include this in the previously discussed PE schema change and likewise include it in the update to all modules.
Quite curious that this didn't come up in CARML over the last years to my knowledge. @tyconsulting I take it you haven't seen this issue before?

[tyconsulting]

Hmm that's new. A batchSize would surely solve the issue - but it's a shame given that it slows down the deployment. Ah well... @segraef I guess we could include this in the previously discussed PE schema change and likewise include it in the update to all modules. Quite curious that this didn't come up in CARML over the last years to my knowledge. @tyconsulting I take it you haven't seen this issue before?

No this is the first time I'm seeing this. I wonder if it's depended on the resource provider. but This is the first time I'm creating 2 PEs for Databricks via this loop. Both PEs have the same group Id but connected to different vnets. Also I wonder if has anything to do with the failure being the same group Id?

[segraef]

Can you give as your deployment code for reference @tyconsulting ?

[tyconsulting]

Can you give as your deployment code for reference @tyconsulting ?

I'll share with you privately tomorrow. Massive power outage here in VIC right noe. I'm on my mobile.

[AlexanderSehr]

Hey @segraef,
I guess the snippet in the description should do the trick:

@batchSize(1) 
module workspace_privateEndpoints 'br/public:avm/res/network/private-endpoint:0.3.3' = [for (privateEndpoint, index) in (privateEndpoints ?? []): { 
 (...)
}

I'd suggest we go about this in 2 steps. Let's first add a second PE deployment to each module in a PR and trigger the pipelines for testing. If consistentpy failing, we can add the batch size to all modules. If not it may only be an issue in some, which we then update in the same PR. Thoughts?

[segraef]

@tyconsulting , just for due dilligence may I have your used deployment code to reproduce the error?
We're currently testing in segraef#7 all multi-PE services and up to 7 concurrent PE deployments without batchSize(1) and no issues so far.

https://github.com/segraef/bicep-registry-modules/actions/runs/8016895436/job/21899816418

[tyconsulting]

Yeah I'll ping you on Monday and give you access to my databricks code in my ado project

[segraef]

I replicated the error: Call to Microsoft.Databricks/workspaces failed. Error message: Workspace update could not be completed because it has been updated by another process. (Code: ConcurrentUpdateError) It seems databricks/workspace-specific, all other services with concurrent PE deployment run fine (#1070).

[segraef]

Testing with batchSize(1) now: https://github.com/segraef/bicep-registry-modules/actions/runs/8030547706

[segraef]

Works with batchSize(1): https://github.com/segraef/bicep-registry-modules/actions/runs/8030547706/job/21945466349

[AlexanderSehr]

Dear @microsoft-gitthub-policy-service, I know and the PRs are open. You may stop now

[segraef]

Implementation in progress, see linked PRs.