Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Private link support for Workload Profiles #867

Open
torosent opened this issue Aug 9, 2023 · 61 comments
Open

Private link support for Workload Profiles #867

torosent opened this issue Aug 9, 2023 · 61 comments
Labels
Networking Related to ACA networking roadmap This feature is on the roadmap

Comments

@torosent
Copy link
Member

torosent commented Aug 9, 2023 •
edited by simonjj
Loading

We are working on Azure Front Door support for this next
@torosent torosent converted this from a draft issue Aug 9, 2023
@microsoft-github-policy-service microsoft-github-policy-service bot added the Needs: triage 🔍 Pending a first pass to read, tag, and assign label Aug 9, 2023
@torosent torosent added roadmap This feature is on the roadmap Networking Related to ACA networking and removed Needs: triage 🔍 Pending a first pass to read, tag, and assign labels Aug 9, 2023
@mortenf1984
Copy link

Any new information on this? It is needed for FrontDoor support with Consumption + Dedicated and internal vNet only

@nad-au nad-au mentioned this issue Sep 20, 2023
Closed
@trylvis
Copy link

trylvis commented Oct 23, 2023

Any estimates on when this will become available?

@wsloth
Copy link

wsloth commented Nov 6, 2023

Looking for a status update on this as well. I'm aiming to run Azure Container Apps behind an Azure Firewall with UDR's, which leaves me no choice but to use Workload Profiles (serverless only would be fine, but it doesn't support UDR's according to this doc).

Workload profiles luckily also supports a serverless variation, but now I'm struggling on how to set up some form of inbound internet connectivity, ideally using Azure Frontdoor. The one guide from Microsoft which details on how to set this up assumes the exposed loadbalancer is called kubernetes-internal, which is not applicable to the Workload Profiles load balancer as it's called capp-svc-lb and is an IP-based backend pool, which is not supported by the Private Link Service.

image

@mortenf1984
Copy link

Can we please get a update on this?

@mortenf1984
Copy link

Are there any new information on the progress to get PLS support to CA with internal LB?

@tasdflkjweio
Copy link

Can we get an update on this? This seems like basic functionality that's missing.

@alhardy
Copy link

alhardy commented Jan 23, 2024

I am wanting to migrate from consumption only to dedicated since I need:

  • UDR support
  • increased Cores per replica

I don't think there are plans for a migration from consumption only to dedicated?

I am using front door for ingress to my consumption only plan, I cannot migrate due to lack of support for private links i.e. connecting front door where apis are on the same domain

@jjindrich
Copy link

My customer has same problem. Any estimates ?

@MrImpossibru
Copy link

+1 waiting for the implementation.
P.S. There're some workarounds if needed: 1) use an Application Gateway, which supports a private endpoint (expensive), 2) custom forwarder based on VMSS and a HAProxy + a Standard Load Balancer + Private Link Service (requires some work to do)

@mitchdowd
Copy link

I'd like to use workload profiles to have NAT gateway support, but then I can't have the environments private which is a big downside.

@rambo108
Copy link

Any update on when the fix will be ready for this?

@jackcaplin
Copy link

Is there an update or ETA?

@jsheetzmt
Copy link

Also looking for an update

@falsadoo
Copy link

Hi all can we get an eta for this?

@mortenf1984 mortenf1984 mentioned this issue Mar 21, 2024
Open
@cachai2
Copy link
Collaborator

cachai2 commented Mar 21, 2024

Hi folks, we are targeting end of Q2 CY2024 for this feature.

@KaiWalter
Copy link

KaiWalter commented Mar 22, 2024 •
edited
Loading

So CY = Calendar Year, hence Apr-Jun I assume. And when @cachai2 @paulyuk do you think, we have this for Azure China / 21vianet? We cannot proceed with our Service Fabric migration to ACA as we will wait for a clean Private Link solution in China.

@ardeshir
Copy link

This is the error we get when attempting to build one directly:

Image

@NemSimpraga
Copy link

It's mid May, any update on this?

@anthonychu anthonychu mentioned this issue May 31, 2024
Open
@chinadragon0515
Copy link
Member

chinadragon0515 commented May 31, 2024 •
edited
Loading

We are ready for private preview, if you are interested in private preview, can you help to fill this form https://forms.office.com/r/fuhKa4UMy6
We will whitelist your sub for private preview.
Public preview still need some time.

@vandanchev
Copy link

We are ready for private preview, if you are interested in private preview, please send you subscription id to acasupport(at)microsoft(dot)com and we can whitelist your sub for private preview. Public preview still need some time.

Would this be available for private preview in China also?

@ardeshir
Copy link

ardeshir commented May 31, 2024 via email

@NemSimpraga
Copy link

NemSimpraga commented Jun 15, 2024 •
edited
Loading

Doesn't this depend on how you created the Container App Environment? Do you have a Workload Profile with ingress set to internal only? Sent from Outlook for Androidhttps://aka.ms/AAb9ysg
…
________________________________ From: Nemanja Simpraga @.> Sent: Friday, June 14, 2024 5:47:00 PM To: microsoft/azure-container-apps @.> Cc: Jack Caplin @.>; Comment @.> Subject: Re: [microsoft/azure-container-apps] Private link support for Workload Profiles (Issue #867) NOTICE! This email was sent from outside Paxton. Please do not click links or open attachments unless you recognise the sender and know the content is safe. I have been playing with this functionality, and I have to say there are some very weird points to consider about the implementation around this feature. Primarily, how the public availability of the application is influenced by the creation of a private endpoint. Example: * Created a private endpoint in my VNet for my Container App Environment * Connected via VNet Gateway into the same VNet where the private endpoint is * I can see that when calling the FQDN of my Container App deployed to that Environment, it resolves to the private IP of my environment's private endpoint and I can successfully connect to my application * When not connected to the VPN, the FQDN of my Container App resolves to a random Azure public IP * importantly, not the actual public IP of the deployed Container App Environment! * this means my application is not available by resolving its FQDN, if I am not connected to my VPN (and configured private DNS resolution) Here's the kicker: I can still reach the Container App over the public IP of the Container App environment! All I have to do is issue an HTTP request directly to the public IP (not the FQDN) of the Container App while setting the "Host" header to the FQDN of my deployed Container App, and I will be able to successfully reach the Container App. This seems like a very weird setup as the application is not completely private (which we can see from the fact that it's available over its public IP), but it is also not available over the public internet when trying to resolve its actual FQDN. What I expected to get after creating a private endpoint for the Container App Environment: * a way into the Container Apps of the Environment via the private network the private endpoint is in * by default, the Container Apps would stay publically available over their FQDN if the access was not specifically restricted to a private IP range via the Container Apps ingress IP restriction What we get is a partial solution: public access is restricted only on the DNS level for the Container Apps since their FQDNs won't resolve to the public IP of the Container App Environment. But, crucially, they will stay available by directly accessing the public IP with a properly set "Host" header. This does not make sense if you ask me, hope it helps in figuring our which direction this feature needs to go. — Reply to this email directly, view it on GitHub<#867 (comment)>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A2AWLZQZBZDXCUNGZERXMOTZHMNAJAVCNFSM6AAAAAA3KJBMG6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRYGQYTCMRXHA. You are receiving this because you commented.Message ID: @.> Jack Caplin Chief IT Architect [https://public.paxton-access.co.uk/public/emailfooterimages/image001.png]https://www.paxton-access.com/?utm_source=staff-signature&utm_medium=email&utm_campaign=staff-signature Phone: 01273 811011 Mobile: 07534563400 Web: www.paxton-access.com<https://www.paxton-access.com/?utm_source=staff-signature&utm_medium=email&utm_campaign=staff-signature> Email: @.@.> Paxton House, Brighton, BN1 9HU Find us on the maphttps://goo.gl/maps/9vNLzgN1qR1zNg9A9 [https://public.paxton-access.co.uk/public/emailfooterimages/image002.png] https://www.facebook.com/PaxtonAccessLtd/ [https://public.paxton-access.co.uk/public/emailfooterimages/image003.png] https://www.linkedin.com/company/paxton-access-ltd/ [https://public.paxton-access.co.uk/public/emailfooterimages/image004.png] https://twitter.com/paxtonaccess [https://public.paxton-access.co.uk/public/emailfooterimages/image005.png] @./> [https://www.paxton-access.com/wp-content/uploads/2024/04/Tech-Tour-2024-%E2%80%93-Email-Signature-1.png]https://www.paxton-access.com/tech-tour/?utm_source=emailsig&utm_medium=email&utm_campaign=techtour&utm_id=techtour

That is true, if you create a Container App Environment with properties.vnetConfiguration.internal as true, the L4 LB you get with the environment has a private IP, and there's no way to reach it directly from the public. But in such Container App Environments, you mostly don't really have a need to create a private endpoint. Perhaps to expose your Container Apps privately to VNets other than the one you integrated the Container App Environment with.

My issue is primarily with ACA Envs with public IPs (i.e. properties.vnetConfiguration.internal = false). If creating a private endpoint will make the FQDNs non-routable publically, then the public IP should not accept internet traffic at all, not let in traffic if someone is pointing calls to the public IP of you environment directly.

The better option, if you ask me (and no one is : ]), is leaving the internet route intact even with a created private endpoint. That is, the FQDNs resolve to the actual public IP of the environment when called from the internet, but when the FQDNs resolve from a VNet, they will resolve to the private IP of the endpoint. This is similar to other private endpoint implementations.

Example: the FQDN of an Azure Flex Postgres server will resolve to the private IP of a private endpoint when the DNS resolution call comes from inside a VNet, but will resolve to the public IP of the server when it comes from the internet. Then on the server itself you can configure firewall rules for the public IP, i.e. let in whatever you want from the public, while private access is handled by the private endpoint.

Container Apps themselves have IP restriction rules, which might be used similarly as a Postgres Flex server firewall rule.

@jackcaplin
Copy link

jackcaplin commented Jun 15, 2024 via email

@NSimpragaVolur
Copy link

NSimpragaVolur commented Jun 15, 2024 •
edited
Loading

I've been using Application Gateway as interim hops.

But if you're using AppGw for interim hops between Front Door and your internal Container App environment, then you don't need the private endpoint. You just put your AppGw in the same VNet as your Container App Environment and route traffic to its IP, which is already private anyways.

Only if your AppGw is integrated into a VNet other than the one your Container App Environment is, private endpoint becomes useful. Which is what I meant by:

Perhaps to expose your Container Apps privately to VNets other than the one you integrated the Container App Environment with.

@jackcaplin
Copy link

jackcaplin commented Jun 15, 2024 via email

@NSimpragaVolur
Copy link

But with the private endpoints, I no longer need the app gateways. Until private endpoint support for workload profiles, they were the only method (apart from app proxies) to integrate front door. I have now removed the app gateways. One per region. Less infrastructure to maintain and less cost. Sent from Outlook for Androidhttps://aka.ms/AAb9ysg
…
________________________________ From: Nemanja Simpraga @.> Sent: Saturday, June 15, 2024 3:12:58 PM To: microsoft/azure-container-apps @.> Cc: Jack Caplin @.>; Comment @.> Subject: Re: [microsoft/azure-container-apps] Private link support for Workload Profiles (Issue #867) NOTICE! This email was sent from outside Paxton. Please do not click links or open attachments unless you recognise the sender and know the content is safe. I've been using Application Gateway as interim hops. But if you're using AppGw for interim hops between Front Door and your internal Container App environment, then you don't need the private endpoint. You just put your AppGw in the same VNet as your Container App Environment and route traffic to its IP, which is already private anyways. Only if your AppGw is integrated into a VNet other than the one your Container App Environment is, private endpoint becomes useful. Which is what I meant by: Perhaps to expose your Container Apps privately to VNets other than the one you integrated the Container App Environment with. — Reply to this email directly, view it on GitHub<#867 (comment)>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A2AWLZVCW2RDTJRALDPRICLZHRDWVAVCNFSM6AAAAAA3KJBMG6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRZG4ZTMMZSG4. You are receiving this because you commented.Message ID: @.> Jack Caplin Chief IT Architect [https://public.paxton-access.co.uk/public/emailfooterimages/image001.png]https://www.paxton-access.com/?utm_source=staff-signature&utm_medium=email&utm_campaign=staff-signature Phone: 01273 811011 Mobile: 07534563400 Web: www.paxton-access.com<https://www.paxton-access.com/?utm_source=staff-signature&utm_medium=email&utm_campaign=staff-signature> Email: @.@.> Paxton House, Brighton, BN1 9HU Find us on the maphttps://goo.gl/maps/9vNLzgN1qR1zNg9A9 [https://public.paxton-access.co.uk/public/emailfooterimages/image002.png] https://www.facebook.com/PaxtonAccessLtd/ [https://public.paxton-access.co.uk/public/emailfooterimages/image003.png] https://www.linkedin.com/company/paxton-access-ltd/ [https://public.paxton-access.co.uk/public/emailfooterimages/image004.png] https://twitter.com/paxtonaccess [https://public.paxton-access.co.uk/public/emailfooterimages/image005.png] @./> [https://www.paxton-access.com/wp-content/uploads/2024/04/Tech-Tour-2024-%E2%80%93-Email-Signature-1.png]https://www.paxton-access.com/tech-tour/?utm_source=emailsig&utm_medium=email&utm_campaign=techtour&utm_id=techtour

Really? I am really curious how you send traffic from Front Door to the private endpoint directly. You usually do it via Private Link Service and the Container App's L4 load balancer with a non IP-based backend pool, which is only applicable for Consumption-only environments. Workload profiles still have the IP-based backend pool L4 LB, so I don't understand how a private endpoint helps overcome this issue?

@dyamo
Copy link

dyamo commented Jul 10, 2024

Hi folks, we are targeting end of Q2 CY2024 for this feature.

Any updates?

@hbuckle
Copy link

hbuckle commented Jul 11, 2024

We've been enabled for the preview and I can confirm that this does let you create a private endpoint attached to the container app environment (so you don't have to create a private link service on the internal load balancer).

However there is still the missing piece that Front Door does not support adding a private link origin attached to the container app environment, which I think is what most people here are after.

@chinadragon0515 - is Front Door going to be updated with origin support for direct private endpoint connectivity to a container app environment?

@chinadragon0515
Copy link
Member

  1. For public IP access even native private endpoint enable, there is new property introduced which is PublicNetworkAccess, you can set to false if you want full disable public access, we will update the private preview document we sent out.
  2. For Azure Front Door integration with native private endpoint support, we are still working with Azure Front Doot team, we do not have a target date to share now.
  3. For Azure front door integration via PLS on internal L4 loadbalancer for workload profile environment, we are also working on this. This is only available in east asia for private preview now, if anyone want to try in east asia, let me know, I can whitelist for you. For this feature private preview in more regions, stay tune, we are working with partner team on this and as we have several service need deployment, still need couple months. I will update here when we are ready to open more regions for private preview.

@delarooster delarooster mentioned this issue Jul 26, 2024
Closed
@masonhuemmer
Copy link

Any update from the Azure Front Door team?

@vmarcella
Copy link
Member

Also requesting an update on this, as my team in the process of migrating a public consumption aca environment to an internal workload profile aca environment and want to expose the container app to the internet through AFD + PLS.

@simonjj
Copy link
Collaborator

simonjj commented Aug 22, 2024

Tentatively aiming for this to become available by the end of the year.

@Expecho
Copy link

Expecho commented Sep 11, 2024

  1. For public IP access even native private endpoint enable, there is new property introduced which is PublicNetworkAccess, you can set to false if you want full disable public access, we will update the private preview document we sent out.
  2. For Azure Front Door integration with native private endpoint support, we are still working with Azure Front Doot team, we do not have a target date to share now.
  3. For Azure front door integration via PLS on internal L4 loadbalancer for workload profile environment, we are also working on this. This is only available in east asia for private preview now, if anyone want to try in east asia, let me know, I can whitelist for you. For this feature private preview in more regions, stay tune, we are working with partner team on this and as we have several service need deployment, still need couple months. I will update here when we are ready to open more regions for private preview.

@chinadragon0515 I understand that coordinating work with the Front Door teams makes it difficult to give an estimation on the issues above but is there anything you can share? Is there actively being worked on?

@jsheetzmt
Copy link

Any possibility of getting this implemented in the mean time? Seems like a low hanging fruit while waiting for proper FD -> container app over private link. #120

@asmild
Copy link

asmild commented Oct 11, 2024 •
edited
Loading

Any updates about private links?
We are unable to use the consumption profile since the Container App's outbound traffic needs to pass through a NAT gateway with a whitelisted IP. Additionally, we need to establish a private link to the application due to internal policy restrictions, such as splitting VNETs and the inability to use peering.

@hen-den
Copy link

hen-den commented Oct 11, 2024

Are there any updates on the progress of this feature? It would be great to understand when we can expect to leverage this in our environments.

@ivanl-out
Copy link

We are in the same boat. We need both UDR and Front Door / private link support. Neither the Consumption-only (v1) nor the Workload Profile (v2) environment types support both features. Are there any updates that can be shared?

@andreaslarssonwalr
Copy link

We are in the same boat. We need both UDR and Front Door / private link support. Neither the Consumption-only (v1) nor the Workload Profile (v2) environment types support both features. Are there any updates that can be shared?

v1 has support for Private Links. That's how we are using it now. But the downside is that you cannot create any v1 Container Envs via the portal anymore (it's possible in bicep/cli), it's only WP enabled ones.

@ivanl-out
Copy link

We are in the same boat. We need both UDR and Front Door / private link support. Neither the Consumption-only (v1) nor the Workload Profile (v2) environment types support both features. Are there any updates that can be shared?

v1 has support for Private Links. That's how we are using it now. But the downside is that you cannot create any v1 Container Envs via the portal anymore (it's possible in bicep/cli), it's only WP enabled ones.

Yes but v1 does not support UDRs. We're looking for an environment type that supports both UDRs and Front Door / private link.

We in fact did use UDRs with v1 (before v2 environment types were even generally available) because unofficially they do work, but got told by Microsoft support that this is not a supported scenario and that our v1 environments can break at any time if we use them.

@simonjj simonjj mentioned this issue Oct 25, 2024
Closed
@rybal06
Copy link

rybal06 commented Nov 7, 2024

Any considerations for enterprise DNS integration? We have had to do an ugly workaround to ensure all our clients can resolve the domain name of the container apps environment. Ideally this solution will be compatible: https://learn.microsoft.com/en-us/azure/cloud-adoption-framework/ready/azure-best-practices/private-link-and-dns-integration-at-scale

Today, we have a DINE policy which provisions an Azure container app Private DNS zone for each container app environment; and then a policy which finds any Private DNS zones which are named for container apps, and does a deployIfNotExists to link the private DNS zone to our name servers vNet.

@KaiWalter
Copy link

@rybal06 we implemented something like https://dev.to/kaiwalter/private-linking-and-port-forwarding-to-non-azure-resources-5h0f to forward DNS requests to our corp DNS. @syneex is currently evaluating whether Azure Private DNS Resolver can replace the VMSS based DNS forwarding.

@syneex
Copy link

syneex commented Nov 8, 2024

@rybal06 we are successfully using the Azure Private DNS Resolver to forward internal domains to our corporate DNS (Outbound Endpoint). I think for your case, the inbound endpoint of Azure Private DNS Resolver would be suitable as they are intended for the ingress to Azure -> Inbound Endpoints.

For that to work, you need to provide the Private DNS Zone for the respective container apps environment anyways...

@rybal06
Copy link

rybal06 commented Nov 8, 2024 •
edited
Loading

I have no issues with hybrid DNS and private endpoints for 50+ other services in Azure using this pattern

My last comment was not clear enough - my question was mostly about whether Microsoft's pattern for "private link and dns integration at scale" will be tested and supported with Azure container apps once private endpoint support becomes GA.

Essentially, whenever any team deploys a private endpoint; using a DeployIfNotExists policy; we evaluate the group ID parameter and parent resource type via policy and automatically link the private endpoint to the appropriate centralized DNS zone in our tenant.

The reasons for this?

  1. Workload teams do not need to be concerned at all with DNS. It is fully taken care of for them by the platform. They don't need to create any DNS records; any private DNS zones; or any tickets for outside teams.
  2. DNS lifecycle management is managed by the platform. When the private endpoint is deleted, Azure automatically deletes the record in the centralized private DNS zone.
  3. We have enterprise-wide domain resolution for azure resources. It doesn't matter if a client is hosted on-premises, AWS, OCI, etc.. because we have DNS forwarding in place between all environments, DNS works automatically across all of them for our teams. The key to this working is having centralized azure private DNS zones which are already linked to the vNET which hosts our recursive, forwarding name servers.
  4. There is no manual intervention needed with private endpoints are provisioned or destroyed.
  5. Workload teams require no permissions on any centralized / shared DNS infrastructure. Azure policy performs all changes on behalf of workload teams.

If I could have a centralized DNS zone in my tenant for privatelink.<REGION_NAME>.azurecontainerapps.io; and an azure policy to detect any private endpoints which are not linked to it, using DeployIfNotExists; assuming the groupID (aka sub resource name) is given on the private endpoint; this solution should work. I have some doubts though because Azure Container Apps uses wildcard(*) DNS records per container app environment rather than A records and have not tested it yet.

Today, my workload teams have to care about DNS when using container apps. They must provision a unique private DNS zone matching the azure-assigned domain name of the container app environment and create the wildcard DNS A record. Our policy detects any *.<REGION_NAME>.azurecontainerapps.io private dns zones and links them automatically to our name server vNET. The problem with this is that there is less validation of the individual A records because it's a decentralized zone that each workload team can modify; and removal of those private dns zones by workload teams fail because they do not have permissions to remove the vNET link from per container app environment private DNS zone. I cannot give each workload team this permission to centralized components.

This would be solved if private endpoints for container app environments works with this pattern: https://learn.microsoft.com/en-us/azure/cloud-adoption-framework/ready/azure-best-practices/private-link-and-dns-integration-at-scale

@curib curib mentioned this issue Nov 19, 2024
Open
@cachai2
Copy link
Collaborator

cachai2 commented Nov 22, 2024 •
edited
Loading

Hi folks, we're excited to share that we now have public preview of private endpoint support for workload profile environments. We are also supporting private link integration with Azure Front Door in the preview (Currently, adding ACA as a private origin for AFD is available through only the CLI with portal support in progress).

For more information, please visit the:

Please let us know if you have any feedback or run into any issues. Thank you!

@onionhammer
Copy link

@cachai2 exciting, is this only supported for premium front door (the one that costs 10x more than standard)?

@cachai2
Copy link
Collaborator

cachai2 commented Nov 23, 2024

Hi @onionhammer, yes. Azure Front Door only supports connecting to origins through private link for its premium offering. If in the future Azure Front Door enables connecting to origins through private link for standard, we will most likely look to enable that as well.

Here is a link to the Azure Front Door docs where they detail the Azure Front Door Premium requirement.

@onionhammer
Copy link

That's disappointing that this security feature isn't emphasized at Microsoft and supported for standard tier Front Door. Even though we would probably utilize this in production, I don't want to waste money using this on our non-production environments, so we will have to support two entirely separate code-paths.

Also this doesn't appear to be configurable via bicep/ARM, so enabling it is still a pain.

@trylvis
Copy link

trylvis commented Dec 7, 2024

Hi folks, we're excited to share that we now have public preview of private endpoint support for workload profile environments. We are also supporting private link integration with Azure Front Door in the preview (Currently, adding ACA as a private origin for AFD is available through only the CLI with portal support in progress).

For more information, please visit the:

Please let us know if you have any feedback or run into any issues. Thank you!

Is there any Bicep / IaC templates available for this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Networking Related to ACA networking roadmap This feature is on the roadmap
Projects
Azure Container Apps Roadmap
Status: Public Preview (Shipped and Improving)
Milestone
No milestone
Development

No branches or pull requests